AWS for Industries
How Rivian transformed its Autonomy data ingestion with AWS Data Transfer Terminal
Rivian, a leading electric adventure vehicle brand, has built its reputation through a relentless focus on customers, as evidenced by their frequent vehicle software updates to enhance their customer experience. The Rivian R1 Gen 2 electric vehicle features an advanced autonomy platform equipped with 55-megapixel High Dynamic Range (HDR) cameras, five radars, and over 200 trillion operations per second (TOPS) of machine learning compute power. Building on this robust foundation, the Autonomy team at Rivian trains sophisticated models, equipping the Rivian Autonomy Platform with one of the most advanced and impactful sets of autonomous vehicle (AV) features in the industry.
Rivian’s data collection test fleet produces terabytes of data on a daily basis from an array of sensors, cameras, and onboard systems. This data is a critical input for Rivian’s autonomous vehicle development as it fuels machine learning models, and helps validate general system performance and enhances safety. However, its sheer volume and variety presents a formidable challenge to Rivian: how to efficiently upload, store, and process it.
Imagine a test fleet of Rivian vehicles traveling across the United States, each producing massive amounts of data from a multitude of sensors and cameras. While this data holds immense potential for insights, it also poses logistical challenges in routing it to centralized storage in Amazon S3. The variety of data types needed for AV development is extensive, encompassing sensor data, camera data, GPS data, control data, and environmental data.
Uploading data from the vehicle data collection test fleet to cloud storage involves several challenges for Rivian. It requires a reliable and high-speed network infrastructure, including high-speed connectivity, edge computing, data security, and data compression. Consequently, the infrastructure needed for uploading data is often costly and complex to implement, especially in remote or poorly connected areas. The challenge is clear: obtaining the capability to seamlessly transfer data over long distances, regardless of origin.
AWS Data Transfer Services
AWS offers several options for customers to transfer their data into or out of the cloud. One option is AWS Direct Connect, which allows customers to establish a dedicated network connection between their on-premises infrastructure and AWS. This provides a reliable, high-throughput connection and requires customers to provision and manage the network equipment at a fixed physical location. Another option is internet-based data transfer, which is more flexible but can be slower and less reliable, especially for large data sets. The AWS Snowball devices offer a way for customers to physically transport large amounts of data to and from AWS data centers. AWS Snowball is suited for periodic data transfer scenarios and data migration, or in locations with limited connectivity.
Rivian adopted an innovative approach by utilizing the new AWS Data Transfer Terminal service, which is designed to provide customers with a secure, physical location where they can bring their storage devices for high-throughput upload to any AWS endpoint—including Amazon S3 and Amazon EFS—via network bandwidth of up to 400 Gbps. AWS Data Transfer Terminal complements Snowball products by offering a location for faster upload to the AWS cloud, minimizing wait times for engineering teams to access data.
AWS Data Transfer Terminal is suitable not only for one-off data transfers but also for recurring data transfer activities (for example, once per month or multiple times per day). Previously, a customer would have needed to order and provision an AWS Direct Connect port in a colocation facility for such periodic use, with the port and space being idle for significant periods, leading to higher costs. With the AWS Data Transfer Terminal, customers can now make ad hoc reservations or schedule recurring visits, paying only for the duration of their usage.
Before: Test fleet driving to OEM office location to upload data or shipping the SSDs. Now: With AWS Data Transfer Terminal, the vehicles drive to the nearest upload location.
Rivian Implementation – Before and After
As Rivian’s Autonomy team expanded data collection, they recognized opportunities to enhance data ingestion efficiency. While existing methods—such as shipping drives or returning to Rivian sites—were effective, they required vehicle operators to spend time off the road. To further optimize operations and maximize data collection, Rivian began exploring new solutions to streamline the process while keeping vehicles in motion.
AWS Data Transfer Terminal helps solve these challenges with its high-bandwidth, low-latency network and strategically located facilities to help ensure more reliable and rapid data transfer. Collaborating with AWS provides Rivian access to a wealth of expertise and infrastructure, helping facilitate swift expansion into new regions without the overhead of building and managing upload facilities. This accessibility also helps maximizes vehicle uptime by providing convenient upload points in major metropolitan areas. Importantly, AWS Data Transfer Terminal maintains a secure chain of custody, helping to ensure that the data remains protected throughout the ingestion process.
By using AWS Data Transfer Terminal, Rivian’s Autonomy team can focus on its core competencies: data collection and advanced driver assistance systems (ADAS) software development. They offload the complex and undifferentiated tasks of designing, implementing, and maintaining the data upload infrastructure to AWS.
How AWS Helped Support Improved Upload Performance for Rivian
Amazon Simple Storage Service (Amazon S3)—object storage built to retrieve any amount of data from anywhere—provides several powerful tools, each tailored to specific use cases. The three most commonly compared options for data transfer to Amazon S3 are the AWS Command Line Interface (AWS CLI), a unified tool to manage your AWS services; the AWS Common Runtime (CRT) libraries; and s5cmd.
The choice among these tools often depends on specific requirements and use cases. When Rivian began its autonomy data upload journey, the company opted to use the AWS CLI, a simple, scriptable tool that was perfect for quickly getting up and running with Amazon S3. In 2024, with the release of the more capable R1 Gen 2 fleet of vehicles, the company required a solution capable of handling larger data volumes at higher speeds. To address this new challenge they adopted the AWS CRT-based S3 client which delivered remarkable throughput results ranging from 2.6 GBps to 3.2 GBps, constrained only by resource limitations of the current generation data storage devices.
The recent upgrade to Rivian’s data upload mechanism with AWS Data Transfer Terminal helps allow the company to ingest, process and analyze data for its model training, three times faster than their previous system. With the ability to handle terabytes of data generated daily by their ADAS fleet, Rivian can now accelerate the development and refinement of self-driving technologies, such as object detection, path planning, and decision-making algorithms.
James Philbin, VP of Rivian Autonomy & AI, shared the following:
“AWS Data Transfer Terminal empowers Rivian to expedite the development of its Autonomy feature roadmap. It helps enable us to process and train models on collected data approximately three times faster than current methods. Crucially, the data stays under Rivian’s control, eliminating the need to ship SSDs. This capability is vital to our data collection initiatives as we scale.”
Conclusion
Rivian’s journey reflects a thoughtful balance between technological advancement and practical execution. From concept to implementation, the company has navigated the complexities of managing large-scale vehicle data collection while prioritizing flexible, efficient upload methods. As Rivian scales its operations, its experience offers valuable insights into the challenges and solutions in automotive data management and transfer. By collaborating with AWS, Rivian delivers on its mission to help its customers stay adventurous forever, regardless of where the road takes them