Palo Alto Networks secures the Internet of Things with Amazon Redshift

By Fan Zhang, Sr Principal / Architect Engineer Software
By Ran Xia, Director, SW Engineering
By Aaron Romero, Sr Staff DevOps Engineer
By Rich Giuli, Principal Solutions Architect – AWS
By Meena Menon, Sr. Customer Solutions Manager – AWS
By Gagan Brahmi, Sr. Specialist Solutions Architect – AWS

Palo Alto Networks

The increase of vulnerable IoT devices in businesses poses a security threat. Palo Alto Networks Internet of Things (IoT )/OT,powered by Amazon Redshift, protects your network and connected devices from potential cyberthreats by providing visibility, risk assessment, and anomaly detection to prevent both known and unknown threats, and strengthening customers’ security posture. This post discusses Palo Alto Networks’ migration to Amazon Redshift to accelerate IoT Security data warehousing capabilities while improving performance, reducing operational overhead, and reducing overall cost.

Palo Alto Networks, an AWS partner, offers a comprehensive suite of cloud-delivered cybersecurity services to tackle the security risks posed by the IoT explosion. Their robust IoT security solution leverages machine learning and automated discovery to provide visibility into all connected devices on customers’ networks, while enforcing Zero Trust principles with least privilege access and continuous security inspection. IoT Security demands the analysis of tens of petabytes of real-time data. Traditional database systems struggled to scale and support such workloads, leading Palo Alto Networks to explore alternative solutions.

Collaboration of Palo Alto Networks IoT and AWS

The successful transformation of Palo Alto Networks IoT/OT Security product data infrastructure was made possible through their collaboration with AWS. Palo Alto Networks worked closely with AWS Solution Architects through an intensive, multi-faceted engagement. This collaboration included weekly strategy sessions, in-depth design reviews, and hands-on implementation workshops. The regular cadence of meetings, coupled with thorough architectural assessments, enabled Palo Alto Networks to fully leverage the AWS team’s expertise for optimally implementing Amazon Redshift and applying industry-proven best practices.

Additionally, Palo Alto Networks leveraged the Amazon Web Services (AWS) Migration Acceleration Program (MAP) to streamline their cloud migration and modernization journey. MAP provides access to a wide range of tools, training, and expertise that reduced costs, automated and accelerated execution, and minimized risks associated with the migration.

Legacy infrastructure challenges

Palo Alto Networks IoT/OT Security products provide threat prevention through its near real-time analysis of network traffic. This capability is critical because any delay in data processing could result in overlooked vulnerabilities or worse, successful cyberattacks. Traditional database systems, which were previously used, faced challenges in scaling to handle both existing and new workloads efficiently. The reporting database, functioning as a data warehouse, was overwhelmed by the task of processing 2 petabytes (PB) of data from various sources, which negatively impacted its performance and efficiency. Palo Alto Networks’ solution addresses these limitations by enabling rapid, scalable data processing, thereby enhancing threat detection and prevention capabilities.

The challenge lay in processing and analyzing petabytes of real-time device session data for online analytical processing (OLAP). Session data, in this context, refers to time-series information about a device’s communication endpoints, including details on which devices are being contacted and the protocols in use. Traditional database infrastructure proved inadequate for handling such large volumes of time-series data efficiently. Attempts to address this issue through over-provisioning meant additional costs, which was not favorable. The existing database design, which used a single system to store various types of data, created a bottleneck in the data pipeline. This approach limited overall performance to that of the slowest operations, highlighting the need for a more specialized and scalable solution.

As the demand for IoT Security grew, the existing infrastructure pipeline faced several critical challenges that needed to be addressed:

• Performance limitations – As the volume of processed data increased, the read and write performance of the legacy database infrastructure suffered a decline, deteriorating by nearly 900%
• Scalability and cost – The cost of scaling the legacy database to handle higher traffic became expensive, with a 30% increase in costs, making it an unsustainable option
• Operational complexity – The legacy database also struggled to handle large spikes in traffic. Scaling the infrastructure during these high-traffic incidents presented a complex operational challenge, often resulting in management difficulties,
• Functional limitations – The legacy database was not optimized to support real-time OLAP and other modern processing capabilities required for the IoT security use case
• Reliability concerns – The infrastructure exhibited instability, necessitating heightened maintenance and operational interventions. This posed challenges in ensuring seamless performance and placed a significant burden on resources.

To address these challenges, Palo Alto Networks explored alternative solutions and decided to collabrated with the AWS to evaluate Amazon Redshift. The adoption of Amazon Redshift improved Palo Alto Networks’ data infrastructure, delivering measurable benefits in performance, cost reduction, and reliability.

New architecture: Yellowstone with Amazon Redshift

With Amazon Redshift, the Palo Alto Networks team was able to build a new IoT data architecture, code-named Yellowstone. This new system leverages advanced data warehousing and reporting capabilities for Palo Alto Networks’ Cloud-Delivered Security Services . As a result, Yellowstone provides an efficient system for IoT Security. Yellowstone is the data pipeline powered by AWS services.

The architecture of Yellowstone is built around Amazon Redshift, a highly scalable and powerful data warehousing solution that significantly enhances the storage and processing of critical device session data. The columnar storage format and sophisticated query processing capabilities of Amazon Redshift allow for substantial improvements in both data processing and retrieval speeds, enabling near real-time data processing.

Yellowstone splits the pipeline, creating a dedicated path for the device session-related data. The session data is first uploaded to Amazon Simple Storage Service (Amazon S3), where it is staged for ingestion into Amazon Redshift. Leveraging stream-based ETL (extract, transform, and load) services, the data is then seamlessly ingested from Amazon S3 into Amazon Redshift, ensuring minimal latency.

To preserve essential legacy reporting features while modernizing the system, the team implemented a hybrid approach. They integrated the new Amazon Redshift Data Warehouse with the existing legacy reporting database. The frontend APIs were then updated to retrieve data intelligently from either source, based on the specific data requirements of each request. By integrating Amazon Redshift with the established legacy infrastructure, the team created a hybrid solution that capitalized on the strengths of both systems. This approach resulted in a robust and flexible platform capable of meeting the diverse and evolving needs of the IoT Security offering, as illustrated in Figure 1.

Figure 1 – Yellowstone architecture powering OT Security

Outcomes

The adoption of Yellowstone, anchored by Amazon Redshift, has delivered significant improvements and addressed the previously mentioned challenges.

Real-time data aggregations – Amazon Redshift’s robust analytical features have empowered the team to perform complex data aggregations in real-time. This enhancement eliminates the requirement for offline data processing and dumps, marking a significant upgrade from the limitations of the previous infrastructure.

Improved API response times – The new Amazon Redshift-powered data infrastructure has significantly improved the performance and scalability of our critical APIs. As a result, we have observed response time improvements of up to 50% for these APIs, leading to enhanced system efficiency and better user experiences as illustrated in Figure 2 showing the 95^th Percentile (P95) performance improvements. P95 is a performance metric which measures the maximum latency in seconds for 95% of the API calls ordered fastest to slowest.

Figure 2 – P95 API response time improvements

Cost Savings – The team has reduced costs for the IoT Security infrastructure by offloading most data processing to Amazon Redshift, resulting in an overall cost reduction of 40%. Figure 3 highlights the cost savings for one of our major clusters, achieved by reducing the legacy database footprint.

Figure 3 – IoT Security infrastructure cost reduction in 2024

Improved Resilience and Stability – Through optimization and processing of data streams using suitable architectures, we have successfully reduced the amount of self-managed infrastructure by 50%. Amazon Redshift’s ability to maintain read performance, even as the write workload increases, ensures low and stable latency in the data pipeline. As a result, the Palo Alto Networks team can handle variable spikes in traffic without manual intervention, leading to increased efficiency and reduced operational overhead.

Summary

The rapid growth of IoT devices has introduced significant security challenges for businesses. Palo Alto Networks IoT/OT Security solution, powered by Amazon Redshift, provides the visibility, risk assessment, and anomaly detection needed to prevent both known and unknown threats. Through their strategic partnership with AWS, Palo Alto Networks was able to overcome the limitations of their legacy infrastructure and build Yellowstone, a scalable and high-performance solution, and enhance data sharing across departments. This has allowed the Palo Alto Networks team to focus on exploring the potential of generative AI, expanding their IoT security solutions, and introducing new features to better serve their customers. Palo Alto Networks has positioned itself as a leading provider of comprehensive IoT security solutions, empowering businesses to mitigate the risks associated with the proliferation of connected devices.

Amazon offers extensive documentation and resources on Amazon Redshift, visit Getting Started with Amazon Redshift. To learn more about Palo Alto Networks IoT Security offerings, visit Enterprise IoT Security and to try the industry’s most comprehensive IoT security free for 30 days, see the free trial offer.

Palo Alto Networks – AWS Partner Spotlight

Palo Alto Networks is an AWS Security Competency Partner that helps customers accelerate cloud migration initiatives with inline and API-based security offerings that complement native AWS security features. VM-Series prevents threats and stops data loss for workloads and containers on AWS. Prisma Cloud enforces policy, configuration and compliance from build phase through run phase. Cortex XSOAR automates response to security incidents.

Contact Palo Alto Networks | Partner Overview | AWS Marketplace