How Rivian optimized cloud storage for its ADAS and Product Engineering workloads on AWS

Rivian Automotive is an electric vehicle manufacturer founded in 2009 and based in Irvine, California. The company offers five-passenger pickup trucks, seven-passenger sports utility vehicles and electric delivery vans. Rivian was the first company to deliver an electric pickup truck to consumers the R1T, in September 2021. Rivian is committed to sustainability, a goal that is reflected in the cloud systems design with use of managed services that can help minimize environmental impact.

This blog discusses how Rivian worked with AWS to adopt cloud architecture best practices and help optimize storage cost in support of its Advanced Driver Assistance Systems (ADAS) and product engineering workloads.

AWS teams work with customers to optimize workloads and help deliver business value at the lowest price point. To address the storage cost challenges faced by automotive customers related to their ADAS and engineering simulation workloads, Rivian worked with the AWS account and AWS OPTICS (Optimization Intelligence for Cloud Systems) teams. The AWS OPTICS team is a group of AWS subject matter experts who guide enterprise customers through ways to help optimize current and future AWS workloads. The team dives into cost and usage data of AWS services, identifies actionable optimization insights, and consults on developing sustainable strategies for optimization, efficiency and performance awareness across the customer’s organization.

Optimizing Amazon S3 for ADAS workloads

Managing storage expenses is a significant consideration for ADAS workloads due to their data-intensive nature. The continuous stream of data from ADAS sensors, capturing high-definition videos, 3D point clouds, and sensor readings generate large volume of data. As vehicles equipped with ADAS capabilities become more prevalent, the sheer volume of data significantly increases, amplifying the challenge of storage management for automakers.

The ADAS teams at Rivian use Amazon Simple Storage Service (Amazon S3), an object storage built to retrieve any amount of data from anywhere, as their primary data store. Amazon S3 offers the scalability needed to accommodate these massive data flows while maintaining accessibility and reliability at low cost. As Rivian started using advanced sensors and high-resolution cameras in their vehicles, the data stored in Amazon S3 started growing 23 percent month over month, resulting in increased storage costs.

To identify ADAS storage cost, Rivian has adopted near-real-time monitoring of Gigabytes stored per month in Amazon S3 using AWS Cost Explorer—an easy-to-use interface that helps customers visualize, understand, and manage AWS costs and usage. AWS Cost Category is a feature within the AWS Cost Management product suite that helps to group cost and usage information into meaningful categories. Rivian utilizes AWS Cost Categories to allocate cost to ADAS teams in order to analyze the storage cost growth patterns and distribution of data among different Amazon S3 storage classes, as shown in Figure 1.

Figure 1. Amazon S3 Storage Tiering View in AWS Cost Explorer example

Rivian also enabled Amazon S3 Storage Lens with advanced metrics, which delivers organization-wide visibility into object storage usage and activity trends and makes recommendations for customers to review to help optimize costs and apply data protection best practices. AWS analyzed the Storage Lens activity data, and determined that the access patterns for ADAS data storage bucket is unpredictable. Hence Rivian adopted Amazon S3 Intelligent-Tiering – a storage class designed to optimize costs by automatically moving data to the most cost-effective access tier when access patterns change.

With its dynamic data tiering mechanism, S3 Intelligent-Tiering automatically moves objects that are not accessed for 30 days to the Infrequent Access tier, then back to the Frequent Access tier when the data is needed again. Rivian activated S3 Intelligent-Tiering Archive Access for rare access to optimize costs even further. Rivian opted in to both Archive Access tier and Deep Archive Access tier and configured them at the bucket level. In addition to the automation of moving infrequently accessed objects to lower-cost storage tiers, S3 Intelligent-Tiering also transfers objects between tiers without transition fees, which made the service optimal for the ADAS data store bucket. To maximize the saving potential and decrease the maintenance overhead, the Rivian application team modified the application to directly upload objects into S3 Intelligent-Tiering. After adopting S3 Intelligent-Tiering for storing ADAS data, Rivian reduced costs by 30 percent, when compared to the standard tier.

While Amazon S3 intelligent-Tiering can help optimize storage cost for unpredictable access patterns, there are certain scenarios where creating a lifecycle policy might be more cost-effective. S3 Intelligent-Tiering temporarily relocates objects for a minimum of 30 days to the Frequent Access tier, which can incur a higher cost than other cost-effective tiers. Also, if you need more granular access to control the objects transition, then lifecycle policies could be evaluated as well.

Optimizing Amazon EFS for engineering workloads

The product engineering teams at Rivian use Amazon Elastic File System (Amazon EFS), which helps to share file data without provisioning storage, as their primary data store. Rivian ramped up the number of engineering simulations to support new vehicle programs, resulting in 60 percent month-over-month increase in the storage footprint in Amazon EFS.

The AWS teams worked alongside Rivian to understand the company’s access patterns and recommended Amazon EFS Intelligent-Tiering to help optimize costs. Amazon EFS supports Infrequent Access storage classes, which are cost optimized for files not accessed every day, with storage prices up to 92 percent lower compared to Amazon EFS Standard storage classes. Rivian’s data is frequently accessed for the first few days of simulation and rarely accessed afterward, so the company implemented a lifecycle policy to transition files from the Standard tier to the Infrequent Access tier after one day since last access as shown in Figure 2. Rivian also did not transition the files out of the infrequent access tier on first access because the files are rarely accessed, helping Rivian to save 70 percent on Amazon EFS storage costs. EFS now also supports a EFS Archive class that could be evaluated for data accessed a few times a year or less.

Figure 2. Lifecycle management in Amazon EFS

For disaster recovery of the Amazon EFS storage, Rivian uses AWS Backup, a cost-effective, fully managed, policy-based service that help simplify data protection at scale. AWS Backup supports a cold storage tier that is 80 percent less expensive than warm storage. Amazon EFS also supports incremental backup with AWS Backup, facilitating a full restore in disaster situations. Full restore can be attained even if the original (full) backup has reached the end of its lifecycle and been deleted.

Rivian implemented a policy to transition the backups to cold storage after eight days as shown in Figure 3 (it is recommended to configure a retention policy with a warm storage duration of at least one week to avoid creating full backups every day). For example, assume the daily backups are retained in warm storage for one day, and assume that the protected resources are so large, it takes the entire day to complete each backup. AWS Backup would implement your desired retention period of one day and remove your backup from warm storage when the backup job was completed. The next day, AWS Backup would not be able to create an incremental backup because there would be no backup in warm storage. Therefore, AWS Backup would create a full backup every day, increasing the backup cost. Hence evaluate your use case and consult with AWS if needed for guidance on creating a retention policy.

Figure 3. Rule configuration for transition to cold storage in AWS Backup

Conclusion

In this blog post, we saw the strategies that Rivian used to help optimize storage costs using a variety of AWS services. A few actionable tips to help optimize your workloads are below.

• Use Amazon S3 Storage Lens to gain organization-wide visibility into object-storage usage and activity. Check out this blog post to help set up S3 Storage lens in your AWS environment.
• Analyze Amazon EFS access patterns and use the range of storage classes that Amazon EFS offers. Check out this re:invent 2023 video and use EFS Archive tier to optimize EFS costs for long lived data that is accessed a few times a year.