AWS Partner Network (APN) Blog
Behind Securonix’s Unified SIEM Spark Architecture on Amazon EKS: 99.99% Availability Across 10,000+ Concurrent Jobs
By: Sanjeev Kishore Yarnapati, Director of Infrastructure Engineering – Securonix
By: Scott Sundby, Senior Technical Account Manager – AWS
 |
| Securonix |
 |
The Financial Services industry attracts some of the most hostile information security threats—and processing more than 78 million security events per second—system performance isn’t just a benchmark, it’s mission critical.
Securonix, a five-time Gartner Magic Quadrant Leader for Security Information and Event Management (SIEM), delivers a security analytics and operations management system powered by artificial intelligence and machine learning. The system combines SIEM, User and Entity Behavior Analytics (UEBA), and Security Orchestration, Automation and Response (SOAR) and Threat Intelligence capabilities to help organizations reduce cybersecurity threats in near real time.
As Securonix continued its rapid growth, its architecture—built on HBase and Amazon EMR running on Amazon Elastic Compute Cloud (Amazon EC2)—had supported expanding workloads for years. With rising data volumes and an increasingly complex threat landscape, the company saw an opportunity to modernize its infrastructure to deliver even greater scalability, resilience, and operational simplicity. A routine Amazon EMR upgrade in early 2024 surfaced areas where system reliability could be further enhanced, prompting a proactive architectural review to support future performance and growth.
Recognizing the need for a strategic transformation, Securonix turned to AWS Enterprise Support to collaborate on this modernization journey. Together, they worked to create a foundation that could simplify operations, strengthen availability, and help Securonix continue to scale to meet growing customer demand.
“High availability and scalability are fundamental to our platform as demand grows. AWS Enterprise Support worked hand in hand with us to align technical outcomes with our business objectives, helping us capitalize on growth opportunities.”
– Sanjeev Kishore Yarnapati, Director of Infrastructure at Securonix
Modernizing Unified SIEM Spark Workloads with Kubernetes at Scale
After conducting technical deep dives with AWS, Securonix launched a 14-month project to modernize its Spark processing environment using Amazon Elastic Kubernetes Service (Amazon EKS). In addition to AWS Enterprise Support, Securonix also worked closely with the Data on Amazon EKS (DoEKS) team to design a Kubernetes-based architecture optimized for scale, cost-efficiency, and resilience. The migration was intentionally phased, beginning with lower-volume workloads to validate performance, and progressively expanding to support the company’s highest-volume, most complex, and performance-critical workloads.
The resulting system runs on more than 40+ Amazon EKS clusters, each supporting between 20−50 customers. These clusters collectively execute over 10,000 concurrent Spark jobs across 5,000–6,000 Amazon EC2 instances. Key components of the architecture include:
• AWS Graviton for cost-efficient compute
• Karpenter for dynamic node provisioning
• Amazon EC2 Spot Instances for fault-tolerant workloads
• On-Demand Instances for high-SLA processing
• Kubeflow Spark Operator to manage Spark job lifecycles
Security data flows from customer environments through custom publishers to Apache Kafka, and then into Spark for real-time analysis on Amazon EKS. With Karpenter provisioning new Amazon EC2 nodes in seconds, the system scales based on demand while maintaining high availability.
A Collaborative Build with AWS Enterprise Support
Throughout the Spark migration, AWS Enterprise Support played a hands-on role in guiding architecture decisions and solving complex technical challenges. AWS Technical Account Manager (TAM) worked closely with Securonix engineers to whiteboard solutions, evaluate proofs of concept, and run deep dives on orchestration, scaling, and cost attribution.
To support the project, the TAM also brought in AWS specialists. Together, the team helped Securonix implement and tune critical components like AWS Graviton and Karpenter.
“Throughout the project, the AWS team helped us think through everything from scaling strategies with Karpenter to cost attribution and performance tuning with AWS Graviton. Their guidance was detailed and practical—they became a part of our team weighing architecture and implementation trade-offs to give the best experience for our platform and customers.”
– Sanjeev Kishore Yarnapati, Director of Infrastructure at Securonix
Conclusion
Securonix’s modernization effort delivered an approximate of 30% improvement in cost performance. This was driven by the price-performance advantage of AWS Graviton and dynamic node provisioning using Karpenter, with Karpenter contributing to a 3–5 percent improvement in operating efficiency. Other improvements across cost, performance, and operations include:
• 99.99% availability maintained across Spark processing workloads
• 91% faster code deployment, reduced from 3–4 hours to 15 minutes
• 50% reduction in job recovery time following Spark job failures
• Hundreds of enterprise customers migrated with minimal disruption
Freed engineering teams from EMR maintenance, improving velocity and enabling focus on strategic initiatives
The migration also had a positive downstream effect on workloads that remained on Amazon EMR, which benefited from reduced processing load and improved stability. With Spark jobs migrated to Amazon EKS, the load on the legacy environment dropped significantly—creating a more stable upgrade path and reducing the risk of future service disruption. The modernization also laid the groundwork for future product expansion, giving Securonix a reusable architectural model for additional services.
Securonix plans to expand its use of Amazon EKS (Elastic Kubernetes Service) by adding more services to the system. The company is also investigating how to use Amazon Bedrock’s generative AI capabilities with enhanced security features and investigation tools.
“Amazon EKS has greatly simplified our deployment process. “We’re actively searching for additional ways to enhance our methods and expand our capabilities.”
– Sanjeev Kishore Yarnapati, Director of Infrastructure at Securonix
Securonix – AWS Partner Spotlight
Securonix is an AWS Advanced Technology Partner that provides cybersecurity with a Unified Defense SIEM powered by AWS and agentic AI, purpose-built to decide and act across the threat lifecycle.