How Rivian and Volkswagen Technology Group Built Real-Time Vehicle Security with Amazon Kinesis Video Streams

In this blog post, we share how Rivian and Volkswagen Group Technologies (Rivian) partnered with AWS to help improve vehicle security through Rivian’s Gear Guard feature enhancement. By using Amazon Kinesis Video Streams (KVS), Rivian created a more sophisticated real-time video streaming solution that helps enable vehicle owners to have more instant access to live camera feeds from their vehicle cameras on the Rivian mobile application.

Introduction

Rivian, a pioneering electric vehicle manufacturer known for its R1T pickup trucks, R1S SUVs, and Electric Delivery Vans (EDVs), is continuously innovating its automotive solutions. Gear Guard, one of Rivian’s core offerings, is a comprehensive family of features designed to help protect the vehicle and its contents when the owner is away. Initially, Gear Guard utilized on-vehicle cameras and AI algorithms to help detect suspicious human activity, record video, and notify the owner via the Rivian mobile app. These videos and recordings were stored locally on the vehicle. The natural evolution of this vital security feature was to introduce real-time live video streaming from the vehicle directly to the Rivian mobile app. This helped provide owners with immediate visual access during detected incidents.

Figure 1 Rivian Gear Guard Character

The Business and Technical Requirements for Live Video Streaming

Implementing live video streaming for Gear Guard demanded a more robust solution that helped address critical functional, performance, and security requirements. The primary goal of the system is to help enable vehicle owners to view live video streams from their Gear Guard security cameras remotely via an authenticated mobile device. Vehicle owners can initiate this live view on-demand or in response to notifications from the Gear Guard alarm system, including while the vehicle is parked, locked, and unattended.

While developing this new live video streaming service, Rivian’s key functional requirements included:

• Remote live viewing from vehicle cameras, including the truck bed camera.
• The ability to select specific cameras or a viewing mode that rotates views between cameras.
• Generating mobile notifications in response to alarm or motion events, with buttons to select the most relevant camera stream and thumbnails for guidance.
• Simultaneous live streaming and recording of events on internal vehicle storage.
• Support for a wide variety of cellular and Wi-Fi network providers and mobile phone devices.

Rivian’s key performance benchmarks that were crucial for a responsive user experience included:

• Activation time: Less than 5 seconds from request to stream display.
• Stream latency: Less than 1 second from camera capture to mobile display.
• Camera switching time: Less than 1 second between camera selection and display.

Rivian’s key privacy and security requirements included:

End User privacy was a paramount concern for Rivian throughout the design process. A privacy threshold analysis and security threat analysis was conducted. For instance, the Gear Guard application is disabled in factory mode to prevent interference with surround view calibration and unwanted video triggers. In addition, Rivian implemented per session and per day limits on usage to help prevent use of this feature for stalking.

Why Amazon Kinesis Video Streams Was Selected

After evaluating several options, Rivian selected Amazon KVS as its video streaming service since it met all the functional, performance/scaling, and security/privacy requirements. Some of Kinesis Video Streams’ salient features that were critical in the decision-making process included:

• Support for multiple streaming and messaging protocols (web real-time communication (WebRTC), Real-Time Streaming Protocol (RTSP), Stream Transmission Control Protocol (SCTP)).
• Robust signaling infrastructure – Fully managed signaling, STUN and TURN server with support for auto-scaling and dynamic creation of channels for communication.
• Comprehensive Observability – Real-time monitoring of server health and uptime, cost monitoring, integration with Amazon CloudWatch for metrics and alerts and the ability to create custom dashboard for performance tracking.
• Security – Built-in integration with AWS security and other native services that integrate well with Rivian’s security model.
• Extensibility – Supports open standards APIs, e.g. V4 signer URL to generate temporary URL with valid signature and allows vendor-neutral functions in Rivian’s IoT and mobile devices. Algorithms, client-side implementation, and reference SDKs in multiple languages are available.
• Performance – Native WebRTC provides support to help with sub-second latency streaming and automatic scaling to help support millions of concurrent streams along with regional deployment for optimal performance.
• Deep technical collaboration – Close partnership with the Amazon Kinesis Video Streams service team enabled successful SDK integration and SigV4 debugging functionality, helping accelerate development and resolve complex implementation challenges.

System Architecture

The Gear Guard live camera architecture is built around WebRTC, which offers lower latency and supports two-way audio/video, helping enable potential voice interaction from the client to the vehicle in future.

Figure 2 Gear Guard Technical Architecture

Start Streaming Sequence

Key Implementation Highlights

Use of data channel for metrics and events exchange between mobile and vehicle.

The WebRTC data channel was utilized for a more robust and bidirectional exchange of information between the mobile application and the vehicle. This included sending remote commands, such as switching camera requests from the mobile application to the vehicle, allowing users to select different camera views. Conversely, tear down requests were sent from the vehicle to the mobile application, indicating the termination of a streaming session. Furthermore, the data channel helped facilitate the transmission of critical metrics, such as the time taken to start streaming, helping enable performance monitoring and optimization. To help ensure more efficient and better-structured communication, all data transmitted over this channel was formatted using protocol buffers (protobuf).

Vehicle certificate-based authentication and delivery for signaling server credentials

A Go programming language-based cloud service helps authenticate and provide the vehicle and mobile application with v4 signed URL with temporary credentials, TURN and STUN connection details for negotiating SDP offer and establishing ICE hops. The request order is irrelevant to the requester, provided the SDP offers are initiated within 5 minutes. Vehicles are authenticated to service using mTLS, with their identity embedded in the certificate. Mobile applications requests are authenticated using oauth2 JWT and verifies if a user is provisioned for the requested vehicle.

The service uses a multi-region Amazon DynamoDB table to help check if a vehicle’s signaling channel exists, then provides a new Amazon Kinesis Video Streams signaling channel for the vehicle or reuses the existing one. The Amazon DynamoDB record helps set an expiration of 30 days, which is updated for every request.

The Amazon DynamoDB record expiration event triggers a lambda function, which helps tear down the signaling channel if unused for 30 days. This helps tracking of active users of the Gear Guard service and helps reduce cost of provisioned resources. The wss endpoint is used to send SDP offer. Embedded within Sig V4 URL is the ChannelARN, ClientId, Expiration, Security Token among others. The service also includes STUN and TURN servers (UDP, secure UDP and TCP) and credentials.

Figure 3 Gear Guard Live Camera Feed

AWS Region-based signaling server and TURN server allocation:

If the peer-peer connection uses TURN servers, we want to allocate the signaling and turn servers in the same AWS Region where the vehicle is located. This is to help avoid the possibility that both a vehicle and mobile application located on the US West Coast and signaling channel are provisioned on the US East Coast. Here are the combinations.

Figure 4 Gear Guard Regional Deployment

To optimize the implementation, the cloud-service, which runs in AWS Elastic Kubernetes Service (Amazon EKS) deployed in both us-east-1 and us-west-2 regions, uses a custom geolocation service to help identify whether a vehicle is located either east or west of Lebanon, Kansas (39°50′N 98°35′W), which AWS and Rivian identified as the middle of the US (as reflected by the broken line in Figure 4). The service accordingly helps provision Kinesis Video Streams signaling channels in the application co-located region where the vehicle is present.
In the above example (Figure 4) car-01 will be set up with a signaling channel in us-west-2 irrespective of whether the mobile is in the East and West coast. Similarly, car-02 will be set up in us-east-1 region.

Production Learnings (1-Year Retrospective)

After a year in production, the following key insights have emerged:

• Latency: The choice of WebRTC proved effective in helping achieve low latency streaming, a significant advantage over other streaming protocols like Http Live Streaming (HLS).
• Scalability for Multiple Viewers: A key architectural challenge with WebRTC’s peer-to-peer nature is that each client requires a separate SRTP/SRTCP stream with unique encryption keys. This means that if a second client requests a stream from the same vehicle, a new stream must be initiated, limiting the direct sharing of a single vehicle stream to multiple peers. The current design is optimized for one peer-to-peer connection for live viewing, with a maximum of 10 participants supported per signaling channel.
• Cost Management: Optimized signaling channel provisioning and teardown on user requests through the connection service. This ensures that end users not using the feature do not have signaling channels pre-provisioned for them.
• Challenges and Solutions:
  • Binding to network interfaces: AWS SDK for C++ did not support binding to specific network interfaces for socket connections which was essential as Rivian’s streaming APN is different from the default network interface. However, as the SDK is open source and Rivian was able to create a patch for network binding and bind to the streaming interface.
  • Vehicle Wake-up Time: Optimizing the time it takes for a sleeping vehicle to respond to mobile commands and initiate streaming was a critical performance consideration.

Enhancements and Future Plans

Rivian continues to evolve the Gear Guard live cam feature with planned enhancements:

• Next generation user interface updates, including adding more visibility for users into different stages of connection and visibility into vehicle network conditions during streaming.
• Hardening successful sessions with optimizations like adaptive bitrate and ICE restarts, including improving session success rates, which is the most important KPI for this feature. AWS WebRTC SDK does support TWCC for media channels which we can use for adaptive bitrate implementation. The SDK also supports restartIce() which we are planning to use to reconnect around connectivity drops/ network switches.

Conclusion

Rivian’s implementation of Gear Guard live camera with Amazon Kinesis Video Streams showcases a more sophisticated solution for helping enhance vehicle security with owners’ privacy being top of mind (no video recording stored on Rivian infrastructure or the AWS cloud). By strategically using WebRTC for its lower latency and two-way communication capabilities and more seamless integration with Kinesis Video Streams for more robust signaling and secure credential management, Rivian has delivered a powerful real-time video streaming experience.