Dow Jones’s AWS Cloud journey and adoption of Amazon EBS io2 volumes

Dow Jones is a global provider of news and business information, delivering content to consumers and organizations around the world across multiple formats, including print, digital, mobile, and live events. It produces leading publications and products including The Wall Street Journal, America’s largest newspaper by paid circulation; Factiva, Barron’s, MarketWatch, Mansion Global, Financial News, Dow Jones Risk & Compliance, and Dow Jones Newswires.

Dow Jones recently upgraded to the new Amazon EBS Provisioned IOPS (io2) volumes, the latest generation of the Provisioned IOPS SSD-based volumes. We use these new Amazon EBS volumes for our database workloads that power our customer-facing applications, and it’s been great for our team, our products, and ultimately our customers. Dow Jones has produced unrivaled quality content for more than 130 years, and today we are one of the world’s largest global newsgathering operations. We simplified how we deliver availability, and we are seeing improvements in that area, while also being able to provision more IOPS without increasing our spend on storage. This means we further minimize how much money and time we spend ensuring higher performance or availability.

Our experience with Amazon EBS has been consistent with our experience working with AWS and the AWS Cloud. I still remember how amazed we were when we were able to provision multiple petabytes of data for one of our databases in mere seconds with Amazon EBS. In that case, we improved our application availability while at the same time lowering our storage costs. Recently, I couldn’t help but thing about our journey to the cloud and how far we’ve come since. I’m excited that our continued work with AWS has presented the opportunity to share my experience here.

Dow Jones’s cloud journey

These days, organizations upgrade their cloud environment as a matter of course. However, when Dow Jones first started using the cloud at the beginning of the last decade, that wasn’t the case. Back then, we relied on on-premises systems, which could be difficult to scale, especially storage. Security often required robust patching strategies – and we spent a lot of time managing and maintaining legacy technologies. Nevertheless, the Dow Jones Technology Leadership Team saw the cloud’s potential, even before all the bells and whistles of today.

I remember when my manager asked me to join a small team dedicated to figuring out how Dow Jones could best use the cloud. In that moment, I was excited to be a part of what felt like the “future.” The leadership team knew that in order to continue innovating, we needed to continue to modernize our infrastructure and help the team get the skills we needed to become cloud experts. I think that’s what I love most about my work and why I have been at Dow Jones since 1999. I am still growing my expertise every day, and I have the opportunity to try new things and figure out solutions as I go.

We selected AWS to be our first cloud provider. They were knowledgeable and had a great track record of successfully migrating legacy systems to the cloud, including storage. The AWS team was really hands-on and we all learned a lot through the process. We first started migrating internal applications. Our teams worked well together and our first migration was a success – it has been a great partnership ever since and we have migrated a lot of data since then. During the process, we have benefited from the agility, flexibility, and functionality of running on AWS, especially Amazon EBS.

Our use of Amazon EBS

One of the first benefits of Amazon EBS we noticed was how quick and easy deployment was compared to the long procurement and deployment cycles associated with on-premises SAN arrays. Our global news database Factiva contains more than 1.9 billion articles going back over 30 years, and continues to grow every day. With Amazon EBS, we were able to provision multiple petabytes of data for Factiva in seconds. This has greatly reduced the time we need to stand up an application. The pay-as-you-go model also gives us the ability to experiment and validate without commitment. For instance, we investigated several different storage models to arrive at one that optimizes both performance and cost.

Amazon EBS also provides five different volume choices to match the unique price-performance needs of our workloads. We chose General Purpose SDD (gp2) volumes for most of our applications, since they match our need for a broad range of transactional workloads, including dev/test environments, low-latency interactive applications and boot volumes. For business-critical, I/O intensive database applications that need high and consistent performance, such as our clusters, we chose Provisioned IOPS io1 SSD volumes. For our throughput-heavy Hadoop Distributed File System (HDFS) applications, we chose a combination of sc1 and st1 HDD volumes. In the on-premises environment, we had to manage different classes of storage area networks (SANs) to host these applications. For example, SSD SAN for hosting the applications, HDD SAN for HDFS applications, and a Hybrid HDD+SSD SAN for all other applications. It was cumbersome to manage all these different classes of SAN arrays. We love that EBS volume types have abstracted all the underlying complexity, and all we need to think about is how much storage capacity we need and storage performance we need.

Additionally, if we must change our volume type based on application demand, Amazon EBS elastic volumes give us the flexibility to seamlessly change volume type. For example, we can change from an SSD-based volume type to an HDD-based volume type or vice versa without disrupting our applications. A similar migration from SSD SAN to HDD SAN and vice versa was cumbersome and required application downtime.

Finally, unlike on-premises systems where we buy capacity based on forecasted demand, Amazon EBS Elastic Volumes let us scale size or IOPS for a volume online as our needs grow. This has reduced our overall TCO, helping us leverage our resources towards other strategic initiatives.

Migrating to new Amazon EBS Provisioned IOPS volumes (io2)

Six years on, a lot has changed, and we have migrated petabytes of storage to Amazon EBS. We are a cloud-first organization with a highly skilled in-house team, and I recently led our upgrade to the new io2 EBS volumes for our workloads. As usual, our team collaborated with the AWS team to make it happen, and we are enjoying the new improvements.

We run our mission-critical systems on io1 volumes, which grant the performance and consistency required to keep these applications running 24/7. We follow AWS best practice and have a Multi-AZ solution to protect against rare Availability Zone level failure. Even with the Multi-AZ environment, we want to reduce the scenarios where we must failover to the environment in the secondary Availability Zone. This is because the failover to the secondary Availability Zone impacts application availability, and once the application flips over to the secondary Availability Zone, it is operating without a HA pair. This puts the application at a higher risk of downtime if there is a failure in the secondary Availability Zone before the primary Availability Zone environment restores.

Amazon EBS volume failures are extremely rare. Notwithstanding, io2 volumes further reduce the likelihood of an EBS volume failure and the need to failover to the environment in the secondary Availability Zone. In addition, the elastic volume functionality enabled us to migrate all the io1 volumes to io2 without any application downtime.

Conclusion

We are excited about the innovation Amazon EBS has rolled out over the past decade, and where AWS is leading the service. Our team is really happy with the new Amazon EBS io2 upgrade. The cloud may be a routine part of most organizations’ technology stack, but like Dow Jones, it just keeps getting better.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.