reference deployment

Data Lake with SnapLogic on AWS

Automated design, setup, and configuration of hardware and software for building a data lake

This Quick Start builds a data lake environment on the Amazon Web Services (AWS) Cloud in about 15 minutes by deploying SnapLogic components and AWS services like Amazon Simple Storage Service (Amazon S3) and Amazon Redshift. You can build a new virtual private cloud (VPC) infrastructure that’s configured for security, scalability, and high availability, or use your existing VPC infrastructure for the data lake.

SnapLogic is a data integration platform that can connect to any source and deliver data in a format that's suitable for analytical solutions. The SnapLogic Groundplex server is the underlying component that processes data from various sources. Groundplex servers are automatically deployed on AWS to enable you to process the data necessary for building analytical solutions.

This Quick Start is intended for users who want to accelerate their big data initiatives. It includes basic integration patterns with Amazon S3 and Amazon Redshift by using SnapLogic integration connectors and best practices from Agilisium Consulting.

micro focus logo
micro focus logo

This Quick Start was developed by Agilisium Consulting in collaboration with AWS. Agilisium Consulting is an APN Partner.

  •  What you'll build
  •  How to deploy
  •  Cost and licenses
  •  What you'll build
  • Use this Quick Start to set up the following environment on AWS:

    • A highly available architecture that spans two Availability Zones.*
    • A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.*
    • In the public subnets:
      • Managed network address translation (NAT) gateways to allow outbound internet access for resources in the private subnets.*
      • A Linux bastion host in an Auto Scaling group to allow inbound Secure Shell (SSH) access to Amazon Elastic Compute Cloud (Amazon EC2) instances in public and private subnets.*
    • In the private subnets, SnapLogic Groundplex servers in an Auto Scaling group to execute the SnapLogic pipelines.
    • In one of the private subnets, a Redshift cluster to store aggregated data for consumption and analysis.

    * The template that deploys the Quick Start into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.

  •  How to deploy
  • To deploy a data lake with SnapLogic on AWS, follow the instructions in the deployment guide. The deployment process includes these steps:

    1. If you don't already have an AWS account, sign up at https://aws.amazon.com, and sign in to your account.
    2. Obtain a SnapLogic trial license and Groundplex configuration file. You can register for a 30-day free trial.
    3. Download the Groundplex configuration file.
    4. Upload the Groundplex configuration file to a private S3 bucket.
    5. Launch the Quick Start. Each deployment takes about 15 minutes. You can choose from two options:
    6. Test the deployment by validating the Groundplex server status.
  •  Cost and licenses
  • You are responsible for the cost of the AWS services used while running this Quick Start reference deployment.

    There are no additional license costs involved for using the Quick Start.

    The AWS CloudFormation template for this Quick Start includes configuration parameters that you can customize. Some of these parameters, such as instance type, will affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you will be using. Prices are subject to change.

    Tip     After you deploy the Quick Start, we recommend that you enable the AWS Cost and Usage Report to track costs associated with the Quick Start. This report delivers billing metrics to an Amazon S3 bucket in your account. It provides cost estimates based on usage throughout each month, and finalizes the data at the end of the month. For more information about the report, see the AWS documentation.

    You need to provide your own SnapLogic license and Groundplex configuration file. If you don’t already have these, you can request a 30-day free trial license by filling out the registration form on the SnapLogic website. You will receive a unique user ID and password from SnapLogic with a Groundplex configuration file to use during the Quick Start deployment process.