Securonix Achieves 30% Cost Savings Processing 10,000+ Spark Jobs on Amazon EKS

Securonix provides a cloud-based SIEM solution that continually processes terabytes of data to help organizations detect internal threats and security issues. As Securonix’s customer base and data volumes grew, the company saw an opportunity to modernize its Apache Spark workload infrastructure to meet customer demand while optimizing operational efficiency, scalability, and costs.

The company’s DevOps team explored options and identified Amazon EKS as the solution that could provide the necessary flexibility for its containerized workloads. Amazon EKS automates cluster infrastructure management, handling everything from container scheduling to dynamic scaling and compute optimization. The service uses AWS enterprise-grade infrastructure and provides native integration with AWS networking, security, and storage services.

“High availability is a key requirement for our software, along with scalability as our workload increases,” says Tanuj Jain, senior director of engineering productivity at Securonix. “Alongside cost optimization, these were driving factors for us. The modernization using Amazon EKS, AWS Graviton, and Karpenter is helping us achieve those goals.”

Securonix engaged the team at Data on EKS (DoEKS)—an open-source project that provides best practices, guidance, and Terraform examples for deploying scalable data processing workloads on Amazon EKS. As a mission-critical cybersecurity provider, Securonix requires 99.99 percent availability and cannot afford to lose any data points. The company functions as a command center for its customers’ security teams, collecting logs from customer environments through data publishers, processing them using Apache Kafka, and analyzing them with Spark to detect security threats.

To maintain this critical service while modernizing the infrastructure, Securonix’s technical account manager collaborated closely with AWS Enterprise Support, which provides optimized cloud services for enterprises. The modernization effort spanned multiple technical and business domains, requiring a broad approach across cost optimization, security, performance, and orchestration. “The expertise from our technical account manager and AWS specialists was instrumental in helping us design for resilience while tackling technical challenges such as implementing cost allocation in our multi-tenant environment,” says Sagar Nadgeri, DevOps engineer at Securonix.

The modernized architecture spans 40 Amazon EKS clusters that run the latest Kubernetes version, across 5,000–6,000 instances, and the entire architecture processes over 10,000 concurrent Spark jobs. The architecture uses Kubernetes to create isolated environments for each tenant, maintaining data separation and security for customers. The data flow begins with Securonix’s custom data publishers that run at customer sites and collect security logs and forward them to Kafka for ingestion. From there, the data moves to the Spark processing layer on Amazon EKS for analysis.

The architecture also relies on Karpenter, an open-source node-provisioning project for Kubernetes, to scale the Amazon EKS clusters. Karpenter automatically provisions new nodes in response to pending pods, spinning up instances in less than 1 minute. As a result, resources are available exactly when needed.

Securonix implemented a strategic mix of instance types across its environment, using AWS Graviton–based instances throughout the infrastructure. Additionally, the solution employs Amazon EC2 On-Demand Instances, which let companies pay for compute capacity by the hour or second (minimum of 60 seconds) with no long-term commitments. It also uses Amazon EC2 Spot Instances, which run fault-tolerant workloads for up to 90 percent discount compared to On-Demand prices. Securonix allocates Spot Instances to low service-level-agreement (SLA) workloads, where occasional interruptions can be tolerated, and On-Demand Instances to high SLA workloads, which require greater availability.

With that robust infrastructure, Securonix maintained its 99.99 percent availability and improved resilience, reducing recovery times from job failures by 50 percent. Additionally, the new architecture minimized maintenance overhead, so a team that was previously dedicated to managing its Spark application can now focus on strategic initiatives by standardizing on Amazon EKS. Securonix also cut costs by about 30 percent and sped up deployment. “Using Amazon EKS architecture, we can now deploy a code change across our clusters in 15 minutes instead of 3–4 hours, increasing agility,” says Jain.

Securonix’s Amazon EKS architecture provides the flexibility, scalability, and operational efficiency the company needs to support its continued growth while maintaining the high performance that its customers expect. Building on the success of this initiative, Securonix will continue to use Amazon EKS to support new workloads. The company is now migrating additional product components to Amazon EKS and considering options for deploying generative AI workloads on the new architecture.

“Using Amazon EKS, our whole deployment practice is simpler,” says Jain. “We’re now exploring further ways to improve our approach and increase our capabilities.”