Dialog Axiata saves significantly on storage using Amazon S3 Intelligent-Tiering and S3 Storage Lens

The telecommunications industry has undergone major technological changes in recent years, and the extensive data generated by telecom operations demands strategic handling. According to a McKinsey survey, over 50% of telecom companies said they were moving more assets to the cloud, including many business-critical applications such as business support systems (BSS). Telecom companies must prioritize optimization strategies alongside their cloud migration to fully reap the benefits of the transition. Poor cloud financial management can lead to higher operational costs, potentially compromising security and scalability for telecom companies.

Dialog Axiata PLC (part of Axiata Group Berhad) is one of Sri Lanka’s largest Quad Play Telecommunications Service providers. With 17.1 million subscribers, Dialog provides various services such as Fixed-line, Broadband, Mobile, Television, Payment apps, and Financial services. With Amazon Simple Storage Service (Amazon S3), Dialog can store virtually unlimited data while also optimizing storage costs.

In this post, we discuss how Dialog uses Amazon S3 Storage Lens, the S3 Intelligent-Tiering storage class, and other tools to efficiently manage large-scale data on AWS while reducing storage costs. Dialog is able to save $90,000 (or 15%) a year on storage costs by optimizing storage based on access patterns and cleaning up incomplete multipart uploads. Consider leveraging these features and strategies to optimize storage costs as you transition your data to AWS.

Improving customer experience with Amazon S3: Dialog’s strategies

MyDialog App and WOW SuperApp are Dialog’s two major business-critical workloads, with more than 900 TB of data stored in Amazon S3. Dialog prioritized these two workloads for cost optimization, since the data size is expected to grow exponentially in the coming months. Before diving into Amazon S3 cost optimization, it is helpful to first understand more about these workloads and how Dialog uses Amazon S3 to best service its customers.

Data analytics: Dialog taps into the wealth of their customer data stored in Amazon S3. By running analytics on this data, Dialog gains valuable insights into customer behaviors, preferences, and usage patterns. This data-driven approach allows the company to tailor its services to better meet customer needs and expectations.

Network optimization: The data stored in Amazon S3 includes information related to call traffic congestion, geographical locations, and call quality. Dialog can use this data to identify network performance issues, pinpoint areas with congestion problems, and address call quality concerns. By addressing these network issues, Dialog can provide a smoother and more reliable communication experience for its customers.

Service enhancement: By leveraging insights gained from Amazon S3 data, Dialog can make informed decisions to enhance its services. This could involve optimizing network resources, introducing new features or offerings, and improving the quality of existing services. All of this contributes to an improved customer experience.

Analyzing object usage patterns: Leveraging AWS native tools at Dialog

1/ Dialog uses S3 Storage Lens for comprehensive storage monitoring, using free metrics that cover general storage data and use-case-specific insights, such as cost optimization and data protection. They employ this tool to identify buckets lacking S3 Lifecycle rules, address incomplete multipart uploads, and make sure that data-protection best practices such as S3 Replication or S3 Versioning are followed. By doing so, Dialog proactively manages their storage environment, optimizes costs, and enhances data security in alignment with Amazon S3’s best practices, ultimately making sure of efficient and reliable storage operations.

2/ Dialog effectively utilizes the Cost and Usage Dashboards Operations Solution (CUDOS) dashboard, an Amazon QuickSight-based tool that relies on the AWS Cost and Usage Report (CUR) to provide granular, hourly insights into resource-level costs and usage. Within the dashboard, Dialog maximizes the Storage tab’s capabilities, dedicated to tracking storage-related costs and their distribution across different storage tiers over time. Dialog extracts valuable information from this tab, such as cost-per-tier and the impact of object movement between tiers on the unit cost of storage. Additionally, they closely monitor the cost incurred through API operations for their top S3 buckets and identify opportunities for potential migration to other S3 storage classes to optimize costs using the following view of the CUDOS dashboard. This approach helps Dialog maintain a cost-effective and well-organized storage environment.

Figure 1: Dialog’s per-S3 bucket spend by API operation

In addition to employing the tools mentioned earlier for analyzing usage and activity metrics, Dialog successfully identified workloads where a lifecycle policy couldn’t be applied by delving into the inherent nature and characteristics of each workload. These specific workloads have been transitioned to the S3 Intelligent-Tiering storage class. Looking ahead, the storage for these identified applications is directly allocated to S3 Intelligent-Tiering, eliminating the need for an interim step of placing them in S3 Standard storage class before moving to S3 Intelligent-Tiering.

Efficient data management with Amazon S3: Optimizing costs and accessibility at Dialog

Amazon S3 has several different storage classes you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.

Dialog’s use of Amazon S3 is highly adaptive, catering to the varying usage patterns of the stored data. Data with predictable access patterns is efficiently managed through S3 Lifecycle, a feature that optimizes storage costs, performance, and data durability by automatically moving objects to the most appropriate storage class. For example, customer billing data from the MyDialog App, which includes recent monthly bills, sees frequent access, while bills older than three months are accessed less frequently. In these cases, S3 Lifecycle policies are employed to automatically adjust storage classes and cost-effectively manage this data.

Conversely, data collected from mobile signal towers, encompassing valuable metrics such as Call Quality Index, is harnessed by multiple analytics applications and ML models, and its access patterns are less predictable. To address this challenge, Dialog turns to S3 Intelligent-Tiering. S3 Intelligent-Tiering is a storage management feature that automatically moves objects to different access tiers based on their usage patterns. Specifically, after an object has not been accessed for 30 consecutive days, S3 Intelligent-Tiering intelligently moves it to the Infrequent Access tier, which offers cost savings of up to 40%. Furthermore, if an object is not accessed for 90 consecutive days, then it is transitioned to the Archive Instant Access tier, providing substantial cost savings of up to 68%. This dynamic approach optimizes cost while preserving accessibility, making it the ideal choice for data with less predictable usage patterns.

In essence, Dialog’s strategic use of Amazon S3, employing both S3 Lifecycle policies and S3 Intelligent-Tiering, allows them to efficiently manage and store data, thereby optimizing costs while making sure data accessibility aligns with real-world usage patterns.

Analyzing the cost impact of S3 Intelligent-Tiering

Since April 2023, Dialog has been effectively using S3 Intelligent-Tiering as the “default storage class” to optimize storage costs. This implementation has resulted in an increased adoption of S3 Intelligent-Tiering over time. The following graph shows the amount of data in the different S3 Intelligent-Tiering access tiers since Dialog’s adoption of this storage solution in April 2023. As you can see, data starts making its way to the Archive Instant Access tier appears automatically (if eligible data exists) after three months of enabling S3 Intelligent-Tiering (enabled April, with AIA appearing in July), helping to reduce storage costs even further.

Figure 2: S3 Intelligence-Tiering access patterns over time at Dialog

Customers pay a monthly monitoring and automation charge per object stored in the S3 Intelligent-Tiering storage class to monitor access patterns and move objects between access tiers. Between April 2023 and September 2023, Dialog’s monitoring cost for S3 Intelligent-Tiering has been less than 1%. This was made possible through the selection of suitable workloads, ones with large size objects, for S3 Intelligent-Tiering, as shown in the following graph

Figure 3: Amazon S3 Intelligent-Tiering monitoring costs as a percentage of total storage costs

After the adoption of S3 Intelligent-Tiering, Dialog has experienced a substantial decrease in the average storage cost. The following graph illustrates an upward trend in the utilization of S3 Intelligent-Tiering, rising from 0% in March to a notable 57% in September. This increase in adoption has corresponded with a significant reduction in the cost of Amazon S3 storage per gigabyte. In March 2023, the cost was $0.0233 per GB, and by September 2023 this cost had decreased by 15.3% to a more economical $0.0197 per GB.

Dialog anticipates that these cost-saving trends will persist into the future, making S3 Intelligent-Tiering an increasingly attractive and cost-effective solution for Dialog’s storage needs.

Figure 4: S3 Intelligence-Tiering storage cost over time at Dialog

Managing incomplete multipart uploads

In addition to effectively leveraging S3 Intelligent-Tiering, Dialog took the proactive step of cleaning up failed incomplete multipart uploads by following the best practices outlined here. This initiative led to a substantial 25% reduction in Amazon S3 storage usage, significantly optimizing the company’s storage footprint. The cleanup effort was particularly impactful, reducing the total Amazon S3 storage usage from 2.09 petabytes in March 2023 to 1.57 petabytes by June 2023. Specifically, Amazon S3 usage dropped by approximately 5% in April, 10% in May, and 13% in June, with a proportionate decrease in S3 spend for each month.

Figure 5: Amazon S3 storage usage dropped after deleting Incomplete Multipart Uploads

Conclusion

In this blog post, we discussed how Dialog Axiata PLC used S3 Intelligent-Tiering and insights from the CUDOS dashboard and S3 Storage Lens to save $90,000 (15%) a year on storage costs. With insights from S3 Storage Lens directing them to buckets without S3 Lifecycle rules, Dialog was able to put in place a data storage strategy for both predictable (S3 Lifecycle) and unpredictable (S3 Intelligent-Tiering) access patterns to save on storage. Cleaning up incomplete multipart uploads also helped them clean up a significant amount of storage to further reduce costs.

Although the Dialog CloudOps team at Dialog has optimized storage costs, their vision is to set up a FinOps practice and enable every product team to optimize their workloads for cost and performance using AWS services and tools such as S3 Storage Lens and AWS CUR. To support this vision, the AWS team conducted a FinOps workshop, trained multiple Dialog product teams, and helped Dialog establish a practice for continuously monitoring and optimizing workload costs on AWS.

Thank you for reading this post, and hope you learned something new and useful. Don’t hesitate to leave your feedback in the comments section.