re:Invent 2023 Cost Optimization highlights that you were not expecting

At AWS re:Invent 2023, there were over 40 FinOps-related announcements. However, some announcements may not even seem FinOps related at first glance. In this blog, we are going to look at AWS re:Invent announcements that are not specifically targeted at cost optimization but can help you optimize. The announcements highlighted in this blog may not appear as ways to save money initially, but they will have a big impact on your spend if you use them to their full potential. Also, our previous blog “Recap of AWS re:Invent 2023 Cloud Financial Management Product Launch Announcements “summarized the FinOps announcements, check that out if you haven’t already.

Amazon CloudWatch Logging Optimization

CloudWatch had two very exciting announcements at re:Invent 2023. First, there is the CloudWatch Logs Infrequent Access class. Secondly, AWS announced CloudWatch Logs Anomaly Detection and Pattern analysis, which provides insights into key trends in your logs. With CloudWatch Logs, you can eliminate the operational overhead of managing multiple solutions and consolidate all your logging.

Utilizing the CloudWatch Logs Infrequent Access Log Class

In the past, customers only had one log class to choose from in CloudWatch, which was sometimes cost prohibitive for certain log types such as debug logs. Now, with the new CloudWatch Logs Infrequent Access (IA) log class, customers can consolidate all their logs in CloudWatch, while maximizing the value of their logging. The IA log class has a 50 percent lower per GB ingestion price compared to the Standard log class. This blog post dives into how you can use the new IA log class for new log groups.

It is important to note that once a log group is created, you cannot change its existing class from Standard to IA or vice-versa. Therefore, make sure to only use the IA log class for a workload that doesn’t require advanced features provided by the Standard log class. Examples of use cases best suited for Infrequent Access:

Workloads that don’t require Standard log class advanced features such as Embedded metric format (EMF), Live Tail, Logs Anomaly Detection and Pattern analysis.
Debug logs or web server logs, as they are quite verbose and rarely require any of the advanced functionality.
Internet of Things (IoT) devices that send detailed logs that are typically only accessed for after-the-fact forensic analysis.

Alongside this, we wanted to highlight how to use this new class in AWS CloudFormation. When creating a log group, you will define the log class and concepts, such as retention policy to ensure you keep logs only for the length of time your business requires them.

myLogGroup:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupClass: INFREQUENT_ACCESS

Using CloudWatch Logs pattern analysis

In terms of maximizing value from your logs, CloudWatch also announced CloudWatch Logs Anomaly Detection and Pattern Analysis. This is a set of machine-learning powered log analytics capabilities in CloudWatch that you can use to quickly interpret your logs, identify unusual events, and uncover trends and patterns across your logs to find the needle in the haystack. This can save critical time during troubleshooting and help you uncover opportunities to improve your logging overall.

How does this relate to costs? Often times, higher logging costs are a result of ingesting logs that may not be giving you the information you need. Using this feature, you are able to better identify that value/criticality of your logs and select the appropriate logging class (Standard or IA). Here are some examples.

Unhelpful logs – Error logs that just say “Error” without any context or actionable insights. In such situation, you can update your logs to capture the right level of detail to give you insights into your workload
Unnecessary logs – In figure 1 below, you can see that patterns 1 and 2 reflect the same set of data. The first one is a print statement and the second is a logged statement. Removing such logs which provide no additional value can help you save on logging costs.
Debug statements – In figure 1 below, pattern three shows a Debug statement. Debug statements in a steady-state application could be producing more logs than needed. Check with your developers if debug statements in production logs are adding value to their real-time investigations. Otherwise if they are used for after the fact analysis, IA might be a great fit

Figure 1: Amazon CloudWatch Logs Patterns Analysis

When it comes to logs you should always be looking to maximize their value. CloudWatch Logs Anomaly Detection and pattern analysis, which are included within your existing standard log class in CloudWatch, can help you do just that. For more information on logging best practices, check out the Instrumenting distributed systems for operational visibility guide.

With these announcements we hope this inspires you to dive deeper into your CloudWatch Logs. If you want to learn more about the re:Invent announcements checkout this recording from the event. To learn more about how to use CloudWatch Logs Anomaly Detection and Pattern Analysis, check out this blog post.

AWS Config now supports periodic recording

Another common service we hear customers ask questions about optimizing is AWS Config. AWS Config is often seen as must-have in all accounts to ensure you can audit changes. But a highly changeable account can lead to unanticipated costs as all infrastructure changes logged by AWS Config have a cost. The new AWS Config periodic recording captures the latest configuration changes of your resources once every 24 hours. Yes, ONCE! This option will reduce the number of changes delivered to AWS Config and therefore lower the cost. If you are just trying to audit your account, periodic recording maybe better suited for you. However, choose this option only if you are not under a compliance framework, for which you would require continuous recording. Checkout how to record resource configuration changes periodically with AWS Config blog to get you started.

When to use AWS Config periodic recording:

Development accounts: When starting to develop in a new AWS account, infrastructure can be constantly deployed and changed. Save yourself some money by enabling daily recording. Then, when it is in a more static mode, change it over to standard recording.
Non compliance regulated accounts: Most people think of continuous compliance and audit as the same thing but they are different. Continuous compliance requires you to have 24/7 coverage of every change for regulatory requirements (aka standard config), whilst auditing is monitoring what is going on in your accounts. When you don’t need to be governed by compliance, enable periodic recording. Check with what your governance policies are within your organization.

How to see resource level cost information?

So, this last highlight is a little bit of a smuggle. I am going to mention three announcements and it’s all about how to find your resources to optimize. Firstly, there is the new Cost Optimization Hub, giving you resource level optimization suggestions from your billing console. This service aggregates many AWS cost optimization recommendations for you! Next, we have the new level of resource granularity in AWS Cost Explorer. You can now see what resources are driving your costs with a simple filter. Finally, there is the new Multi-account search in AWS Resource Explorer Manager (RAM) which allows you to search for any resource ID across your AWS Organization.

How does these link together? Imagine you notice a resource in your Cost Optimization Hub, or you see it’s driving the majority of your spend, you can copy that resource ID, pop it in your console search box and jump straight to it! These services will speed up any FinOps investigation work massively.

Please note you have to enable these resource level features. All are free, but you have to go in and click “enable” for them to work. Do it today and go see where you can optimize in your organization.

Honorable Mentions

Here are a few highlights for further reading:

And with that, you have my top AWS re:Invent announcements for 2023. If this blog has inspired you to try one out, let me know or share the blog with your teammates.

AWS Cloud Financial Management