AWS for Industries

How Rivian transformed its Autonomy data ingestion with AWS Data Transfer Terminal

Rivian, a leading electric adventure vehicle brand, has built its reputation through a relentless focus on customers, as evidenced by their frequent vehicle software updates to enhance their customer experience. The Rivian R1 Gen 2 electric vehicle features an advanced autonomy platform equipped with 55-megapixel High Dynamic Range (HDR) cameras, five radars, and over 200 trillion operations per second (TOPS) of machine learning compute power. Building on this robust foundation, the Autonomy team at Rivian trains sophisticated models, equipping the Rivian Autonomy Platform with one of the most advanced and impactful sets of autonomous vehicle (AV) features in the industry.

Rivian’s data collection test fleet produces terabytes of data on a daily basis from an array of sensors, cameras, and onboard systems. This data is a critical input for Rivian’s autonomous vehicle development as it fuels machine learning models, and helps validate general system performance and enhances safety. However, its sheer volume and variety presents a formidable challenge to Rivian: how to efficiently upload, store, and process it.

Imagine a test fleet of Rivian vehicles traveling across the United States, each producing massive amounts of data from a multitude of sensors and cameras. While this data holds immense potential for insights, it also poses logistical challenges in routing it to centralized storage in Amazon S3. The variety of data types needed for AV development is extensive, encompassing sensor data, camera data, GPS data, control data, and environmental data.

Uploading data from the vehicle data collection test fleet to cloud storage involves several challenges for Rivian. It requires a reliable and high-speed network infrastructure, including high-speed connectivity, edge computing, data security, and data compression. Consequently, the infrastructure needed for uploading data is often costly and complex to implement, especially in remote or poorly connected areas. The challenge is clear: obtaining the capability to seamlessly transfer data over long distances, regardless of origin.

AWS Data Transfer Services

AWS offers several options for customers to transfer their data into or out of the cloud. One option is AWS Direct Connect, which allows customers to establish a dedicated network connection between their on-premises infrastructure and AWS. This provides a reliable, high-throughput connection and requires customers to provision and manage the network equipment at a fixed physical location. Another option is internet-based data transfer, which is more flexible but can be slower and less reliable, especially for large data sets. The AWS Snowball devices offer a way for customers to physically transport large amounts of data to and from AWS data centers. AWS Snowball is suited for periodic data transfer scenarios and data migration, or in locations with limited connectivity.

Rivian adopted an innovative approach by utilizing the new AWS Data Transfer Terminal service, which is designed to provide customers with a secure, physical location where they can bring their storage devices for high-throughput upload to any AWS endpoint—including Amazon S3 and Amazon EFS—via network bandwidth of up to 400 Gbps. AWS Data Transfer Terminal complements Snowball products by offering a location for faster upload to the AWS cloud, minimizing wait times for engineering teams to access data.

AWS Data Transfer Terminal is suitable not only for one-off data transfers but also for recurring data transfer activities (for example, once per month or multiple times per day). Previously, a customer would have needed to order and provision an AWS Direct Connect port in a colocation facility for such periodic use, with the port and space being idle for significant periods, leading to higher costs. With the AWS Data Transfer Terminal, customers can now make ad hoc reservations or schedule recurring visits, paying only for the duration of their usage.

Before: Test fleet driving to OEM office location to upload data or shipping the SSDs. Now: With AWS Data Transfer Terminal, the vehicles drive to the nearest upload location.

Before: Test fleet driving to OEM office location to upload data or shipping the SSDs. Now: With AWS Data Transfer Terminal, the vehicles drive to the nearest upload location.

Rivian Implementation – Before and After

As Rivian’s Autonomy team expanded data collection, they recognized opportunities to enhance data ingestion efficiency. While existing methods—such as shipping drives or returning to Rivian sites—were effective, they required vehicle operators to spend time off the road. To further optimize operations and maximize data collection, Rivian began exploring new solutions to streamline the process while keeping vehicles in motion.

AWS Data Transfer Terminal helps solve these challenges with its high-bandwidth, low-latency network and strategically located facilities to help ensure more reliable and rapid data transfer. Collaborating with AWS provides Rivian access to a wealth of expertise and infrastructure, helping facilitate swift expansion into new regions without the overhead of building and managing upload facilities. This accessibility also helps maximizes vehicle uptime by providing convenient upload points in major metropolitan areas. Importantly, AWS Data Transfer Terminal maintains a secure chain of custody, helping to ensure that the data remains protected throughout the ingestion process.

By using AWS Data Transfer Terminal, Rivian’s Autonomy team can focus on its core competencies: data collection and advanced driver assistance systems (ADAS) software development. They offload the complex and undifferentiated tasks of designing, implementing, and maintaining the data upload infrastructure to AWS.

How AWS Helped Support Improved Upload Performance for Rivian

Amazon Simple Storage Service (Amazon S3)—object storage built to retrieve any amount of data from anywhere—provides several powerful tools, each tailored to specific use cases. The three most commonly compared options for data transfer to Amazon S3 are the AWS Command Line Interface (AWS CLI), a unified tool to manage your AWS services; the AWS Common Runtime (CRT) libraries; and s5cmd.

The choice among these tools often depends on specific requirements and use cases. When Rivian began its autonomy data upload journey, the company opted to use the AWS CLI, a simple, scriptable tool that was perfect for quickly getting up and running with Amazon S3. In 2024, with the release of the more capable R1 Gen 2 fleet of vehicles, the company required a solution capable of handling larger data volumes at higher speeds. To address this new challenge they adopted the AWS CRT-based S3 client which delivered remarkable throughput results ranging from 2.6 GBps to 3.2 GBps, constrained only by resource limitations of the current generation data storage devices.

The recent upgrade to Rivian’s data upload mechanism with AWS Data Transfer Terminal helps allow the company to ingest, process and analyze data for its model training, three times faster than their previous system. With the ability to handle terabytes of data generated daily by their ADAS fleet, Rivian can now accelerate the development and refinement of self-driving technologies, such as object detection, path planning, and decision-making algorithms.

James Philbin, VP of Rivian Autonomy & AI, shared the following:

“AWS Data Transfer Terminal empowers Rivian to expedite the development of its Autonomy feature roadmap. It helps enable us to process and train models on collected data approximately three times faster than current methods. Crucially, the data stays under Rivian’s control, eliminating the need to ship SSDs. This capability is vital to our data collection initiatives as we scale.”

Conclusion

Rivian’s journey reflects a thoughtful balance between technological advancement and practical execution. From concept to implementation, the company has navigated the complexities of managing large-scale vehicle data collection while prioritizing flexible, efficient upload methods. As Rivian scales its operations, its experience offers valuable insights into the challenges and solutions in automotive data management and transfer. By collaborating with AWS, Rivian delivers on its mission to help its customers stay adventurous forever, regardless of where the road takes them

Asif Khan

Asif Khan

Asif Khan is a Principal Solutions Architect at Amazon Web Services supporting enterprise automotive customers. He has a passion to design, build, and deliver innovative, cost effective and scalable solutions for the automotive industry. Outside of work, he enjoys mentoring young professionals and staying abreast of emerging tech trends by building prototypes.

Aditya Somayaji

Aditya Somayaji

Aditya Somayaji, an Autonomy Systems Manager at Rivian, leads initiatives in Data Collection Architecture and Firmware Integration. His passion for cars, technology, and a sustainable electric future drives his focus on enhancing autonomy features and improving vehicle safety through cutting-edge technologies. Outside of work, Aditya enjoys exploring nature, traveling to new destinations, and honing his photography skills.

Matt Haller

Matt Haller

Matt Haller, a Global Account Representative at AWS, has over ten years of experience in delivering strategic business outcomes to customers.

Nick Nguyen

Nick Nguyen

Nick Nguyen, a Director of Product and Programs for Autonomy at Rivian, where he focuses on enhancing the driving experience for Rivian owners. His work aims to improve safety features and reduce driver stress during various outdoor activities and long-distance travel. Nick's passion for this role stems from his childhood experiences exploring the country on extensive road trips. He is committed to making similar experiences accessible to more people in an environmentally sustainable manner. Outside of work, Nick dedicates time to raising his children, developing Android applications, and pursuing 3D printing projects.

Ajay Paknikar

Ajay Paknikar

Ajay Paknikar, a Principal Customer Solutions Manager at AWS, supports global automotive customers. Ajay is passionate about guiding enterprises in their AWS journey, leveraging the best of AWS to ensure successful business outcomes. As a strategic advisor to client executives, he focuses on cloud adoption and enhancing cloud maturity.

Venkat Devarajan

Venkat Devarajan

Venkat Devarajan is a Senior Solutions Architect at Amazon Webservices (AWS) supporting enterprise automotive customers. He has over 18 years of industry experience in helping customers design, build, implement and operate enterprise applications.