Improving Developer Deployment Times from Weeks to Minutes Using Amazon MSK with FactSet

FactSet provides access to financial data and analytics through its software solutions to help its clients make sound and efficient investing decisions. Founded in 1978, the company delivers financial data, analytics, and open technology to over 7,500 global institutions, including more than 185,000 individual users. Although the company had already been using AWS for other parts of its stack, it had continued to manage its Kafka clusters on premises. FactSet’s observability team, whose primary responsibilities included providing logging for internal stakeholders such as business analytics teams and site reliability engineering teams, wanted to make its Kafka cluster management more time and cost effective. By migrating its Kafka clusters to the cloud, the observability team could reduce wait times and make the experience more seamless for its stakeholders. “We wanted a solution where we didn’t have to manage the infrastructure and instances directly,” says Chris Bertsch, principal software engineer at FactSet. “Finding a fully managed solution for Kafka was an ideal situation for us.”

The Amazon MSK migration took about 6 months, and the company achieved this all in-house without needing third-party support. The team experienced virtually zero downtime during the migration, using Apache Kafka mirroring (MirrorMaker) to mirror the traffic from the on-premises Kafka cluster to Amazon MSK. After a period of running in shadow mode while replicating in the cloud, the actual migration cutover took a mere week. “With numerous internal stakeholders connecting to our Kafka cluster, it was integral that we coordinated a time frame with them,” says Bertsch. “We were happy with the migration time frame.”

Migrating to Amazon MSK has saved time for FactSet’s observability and other internal teams, who had been tediously coordinating among themselves to configure their on-premises Kafka clusters. Now, with full ownership of the stack, the observability team can scale its compute resources at will without having to submit tickets to the infrastructure team. This scalability improves productivity and provides the team with the ability to support an overall DevOps strategy to accelerate the process of providing features, applications, and logs. Similarly, the infrastructure team, which formerly spent its time managing the technological aspects of the on-premises Kafka clusters, now has the bandwidth to focus on other projects. As a result, the company’s stakeholders have felt the benefit as well. “Previously when we had to start sending more logs to different consumers, we would have to wait for 2 to 3 weeks,” says Bertsch. “Now, since we fully manage virtually all aspects through Amazon MSK, we can make those changes in a matter of minutes.”

After migrating its clusters to Amazon MSK, FactSet reduced its operational costs due to the tiered storage feature within Amazon MSK. Previously, with the on-premises Kafka cluster, the company stored hot data, which demands swift retrieval, for 7 days in case backfill and reprocessing were necessary. Using the various tiers of cloud storage in Amazon MSK, the company has gained greater flexibility to differentiate between hot data and more rarely accessed cold data. Hot data is stored in performance-optimized primary storage, whereas cold data is automatically migrated into the low-cost storage tier. In addition, the tiered storage feature is fully managed and does not require provisioning, which means it can scale to virtually unlimited storage. “The tiered storage feature of Amazon MSK has been a game changer for us,” says Bertsch. “It has greatly reduced our costs.” Overall, the company has used this functionality to reduce its costs by 56 percent.

The company has also improved overall availability, which has been crucial to serving its international clientele. Whereas the previous legacy system was located in a single data center, migrating the clusters to the cloud has granted the team the option to deploy its services across different AWS Availability Zones. Furthermore, using Amazon MSK, the company has improved the elasticity of its infrastructure and can scale up in a pinch. “By being on AWS, we can burst and scale at ease as the marketplace changes,” says Bertsch. “It was vital for us to get all of our infrastructure on the cloud, including our Kafka elements.”

Looking ahead, FactSet plans to use the increased availability of its infrastructure across multiple AWS Availability Zones to find growth opportunities for the company.

“The next big thing for us is to be multiregion,” says Bertsch. “Previously, all our Kafka infrastructure was in a single data center, and the latencies between different regions were far too high. But now, the feasibility of deploying Amazon MSK to virtually any region opens up brand-new possibilities.”