Reducing Operational Overhead and Increasing Reliability Using Amazon MSK with Cisco

Founded in 1984, Cisco has grown to be one of the largest technology companies in the world. It provides networking, security, collaboration, cloud management, and other technology solutions to securely connect industries and communities. Among its subsidiaries is Cisco Webex, which offers the Webex suite of web conferencing and videoconferencing applications. Cisco Webex is a collaboration product suite that includes calling, messaging, meetings, polling, webinars, and events. Offered as a single solution through the Webex app in more than 195 countries, the Webex suite is available for subscription to either partial features or the full suite.

Webex’s observability stack consists of an Elasticsearch, Logstash, and Kibana stack and Apache Kafka and handles 2 PB of data a month. Observability points of presence are deployed around the world to provide Webex DevOps teams, developers, incident management teams, and customer success teams with application monitoring data. However, the original stack was composed of self-managed Kafka, which needed constant certificate management, upgrades, and scaling.

As part of an initiative to improve its existing processes, Cisco’s observability team decided to look for a solution to increase the performance of its stack by removing the required maintenance. Shortly after highlighting this potential area of improvement, the team turned to Amazon MSK.

The migration from the legacy cluster to Amazon MSK was seamless. The observability team spent 5 months engineering the migration from self-managed Kafka to Amazon MSK and then implemented the new stack in 6 weeks. Overall, the team saw better retention, reliability, and performance. The entire system is composed of 12 Amazon MSK clusters with a total of 66 brokers. The system seamlessly handles a daily throughput of 147 TB of data and more than 100 billion messages. “We run a globally federated observability stack at scale, and it just works,” says Robert Labrie, manager for observability at Webex. “I’m really proud of what the team accomplished overall.”

More than 4,000 Webex developers, managers, and team members using Cisco Webex’s solution have benefited from the performance boost of the new stack. The self-managed stack occasionally experienced service outages or delays, which diminished the productivity of users. The new stack using Amazon MSK has been consistently reliable, and teams use it to get their observability data in near real time. Using Amazon MSK, Cisco has reduced the data transport time from the backend to the user, so retrieval times are under 10 seconds. With access to near real-time data, the teams provide a more streamlined customer experience by reducing data latency. “If people are trying to troubleshoot a customer issue, they are relying on our observability data,” says Labrie. “We don’t want them to have to wait for minutes, and our solution takes care of that.”

By migrating from the self-managed cluster to a managed service, Cisco’s observability team has been able to boost productivity. With the reliability of AWS infrastructure, the team sustains the high availability needed for centralized observability while reducing operational burdens, like upgrades, certificate roles, and identity management. “We look at the human resources and technical responsibilities we have so that we’re able to see where we can get the biggest payoff,” says Labrie. “Using Amazon MSK, we are able to deploy our human resources strategically into areas where they can provide the most value.”

Cisco has improved the scalability of its observability stack greatly since migrating to Amazon MSK. Previously, the self-managed tooling was not flexible enough to tune clusters to specific workloads. In essence, the observability team had to employ a one-size-fits-all approach and deploy everything on unnecessarily large instances. Now, it no longer has to over-provision its clusters because Amazon MSK automatically scales to match the capacity needed for each workload. Cisco has reduced its operational overhead to virtually zero with cost savings and decreased service outages.

Using AWS managed services has also benefited Cisco in terms of compliance. Cisco Webex runs a global solution, and thousands of different security and compliance controls exist around the globe regarding data privacy and integrity. Because Amazon MSK comes ready with robust security standards and compliance certifications, the team has the flexibility to use it in virtually all environments, including those that require federal compliance, such as the Federal Risk and Authorization Management Program. “Using an AWS managed service is an automatic checkbox,” says Labrie. “We can be confident that the solution has the right level of cryptography or disk encryption because it’s been certified for use in a federally regulated environment.”

Going forward, Cisco’s observability team plans to innovate in the space of big data, using the observability stack to fuel new projects and explorations. One of those projects is looking into distributed tracing, which helps identify issues in application performance that affect the user experience, like bugs, errors, or high latency. The team is also looking for ways to implement AWS Global Accelerator, a networking service that businesses use to improve the availability, performance, and security of public applications. Cisco’s observability team continues to work with other teams within the organization to find innovative ways to use its data.

“By using Amazon MSK, we are realizing efficiencies, driving our costs down, and improving the reliability of our service,” says Labrie. “Simply put, it just works.”