This Guidance demonstrates how to build an Industrial Data Fabric with Palantir Foundry on AWS. Foundry is a software as a service (SaaS) product that harnesses data from various industrial data sources such as enterprise resource planning (ERP), a manufacturing execution system (MES), and data lakes. The data is rapidly integrated with fully automated pipelines and low-code tools that allow you to train and build machine learning models. You can build or customize manufacturing applications supporting everything from shop floor scheduling to a global operations center. Full-featured web applications provide you with real-time visibility, decision-making tools, and the ability to resolve operational decisions.
Architecture Diagram
[Architecture diagram description]
Step 1
Internet of Things (IoT) data from sensors and programmable logic controllers (PLCs) is ingested using AWS IoT Greengrass and sent to AWS IoT SiteWise. You can build assets and dashboards in AWS IoT SiteWise.
Step 2
Create rules in AWS IoT Core to stream time series data to Palantir Foundry using Amazon Kinesis Data Firehose. Foundry data connectors in the AWS Cloud are used to ingest data from the operational technology (OT) data lake stored in Amazon Simple Storage Service (Amazon S3) to Foundry.
Step 3
Foundry has data connectors that can ingest data from other data sources such as enterprise resource planning (ERP), a manufacturing execution system (MES), and data lakes.
Step 4
Data from the connectors is rapidly integrated with fully automated pipelines and low-code tools.
Step 5
The Foundry Ontology harmonizes data from the semantic elements (such as objects, properties, and links), and kinetic elements (such as actions, functions, and dynamic security).
Step 6
Install the Foundry software development kit (SDK) in Amazon SageMaker Notebooks that offers fully managed Jupyter Notebooks. Configure Foundry API keys to access Foundry datasets and Ontology objects.
Step 7
Use datasets from Foundry to train and build machine learning (ML) models in Amazon SageMaker. Models hosted in AWS can be imported or invoked remotely in Foundry workflows.
Step 8
Build or customize manufacturing applications supporting everything from shop floor scheduling to a global operations center. Full-featured web applications provide you with real-time visibility, decision-making tools, and the ability to resolve operational decisions.
Step 9
Operational decisions and results can be written back to the source system, ensuring consistency across Product Lifecycle Management (PLM), ERP, MES, and other key systems.
Step 1
Internet of Things (IoT) data from sensors and programmable logic controllers (PLCs) is ingested using AWS IoT Greengrass and sent to AWS IoT SiteWise. You can build assets and dashboards in AWS IoT SiteWise.
Step 2
Create rules in AWS IoT Core to stream time series data to Palantir Foundry using Amazon Kinesis Data Firehose. Foundry data connectors in the AWS Cloud are used to ingest data from the operational technology (OT) data lake stored in Amazon Simple Storage Service (Amazon S3) to Foundry.
Step 3
Foundry has data connectors that can ingest data from other data sources such as enterprise resource planning (ERP), a manufacturing execution system (MES), and data lakes.
Step 4
Data from the connectors is rapidly integrated with fully automated pipelines and low-code tools.
Step 5
The Foundry Ontology harmonizes data from the semantic elements (such as objects, properties, and links), and kinetic elements (such as actions, functions, and dynamic security).
Step 6
Install the Foundry software development kit (SDK) in Amazon SageMaker Notebooks that offers fully managed Jupyter Notebooks. Configure Foundry API keys to access Foundry datasets and Ontology objects.
Step 7
Use datasets from Foundry to train and build machine learning (ML) models in Amazon SageMaker. Models hosted in AWS can be imported or invoked remotely in Foundry workflows.
Step 8
Build or customize manufacturing applications supporting everything from shop floor scheduling to a global operations center. Full-featured web applications provide you with real-time visibility, decision-making tools, and the ability to resolve operational decisions.
Step 9
Operational decisions and results can be written back to the source system, ensuring consistency across Product Lifecycle Management (PLM), ERP, MES, and other key systems.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
As your operations evolve over time, you can implement changes within this Guidance so that you can establish a continuous cycle of improvement.
With Amazon CloudWatch, you have system-wide visibility into your cloud resources. Configure CloudWatch alarms so that you can collect, monitor, act, and analyze any breaches in the thresholds you set.
The services in this Guidance can also be deployed across multiple Availability Zones to ensure that your applications can withstand the rare, but possible, event of a complete Availability Zone failure.
-
SecurityRead the Security whitepaper
When deploying this Guidance, all data in transit is encrypted using SSL and data at rest is encrypted using AWS Key Management Service (AWS KMS).
This Guidance also uses a SAML-based authentication and authorization standard for any users. Foundry uses role-based access with auditing enabled when accessing services and machines.
-
ReliabilityRead the Reliability whitepaper
To ensure a highly available network topology, this Guidance uses Amazon S3 to store data. Any applications running on Amazon Elastic Compute Cloud (Amazon EC2) are deployed across three Availability Zones to improve availability and fault tolerance.
Foundry implements logging for any customer actions and workflows so that you are notified when thresholds are crossed or significant events occur.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
This Guidance scales the compute nodes that are needed to perform the task to meet the workload requirements of various traffic and data access patterns. In addition, it uses purpose-built storage services, such as Amazon S3, that reduces latency, increases throughput, and is highly scalable.
-
Cost OptimizationRead the Cost Optimization whitepaper
There are various ways this Guidance was designed to reduce cost. Amazon S3 Intelligent-Tiering storage class automates storage cost savings by moving data when patterns change. And, by using Amazon EC2 Instance Savings Plans, you can take advantage of a flexible pricing model that can reduce your on-demand bill in exchange for a one- or three-year hourly spend commitment.
-
SustainabilityRead the Sustainability whitepaper
This Guidance uses AWS Auto Scaling to help monitor applications and adjust capacity so that you maintain performance with only the minimum resources required.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.