reference deployment

Confluent Platform on AWS

Streaming platform built on Apache Kafka for large-scale, distributed environments

This Quick Start automatically deploys Confluent Platform on the AWS Cloud.

Confluent Platform is a streaming platform for large-scale distributed environments, and is built on Apache Kafka. Confluent Platform enables all your interfaces and data systems to be connected, so you can make decisions leveraging all your internal systems in real time.

This Quick Start is for users who are looking to evaluate and use the full range of Confluent Platform and Apache Kafka capabilities in the managed infrastructure environment of AWS. The Quick Start supports two software editions: Confluent Open Source and Confluent Enterprise.

This Quick Start deploys Confluent Platform using AWS CloudFormation templates. You can use the Quick Start to build a new virtual private cloud (VPC) for your Confluent Platform cluster, or deploy Confluent Platform into an existing VPC.

Alternatively, you can use Confluent Cloud, which is a fully managed Apache Kafka as a service on AWS.

confluent-logo

This Quick Start was developed by Confluent, Inc. in partnership with AWS. Confluent is an
APN Partner.

  •  What you'll build
  •  How to deploy
  •  Cost and licenses
  •  What you'll build
  • Use this Quick Start to set up the following Confluent Platform on AWS:

    • A VPC configured across two Availability Zones, if you use the new VPC deployment option. For each Availability Zone, this Quick Start provisions one public subnet. The Confluent Platform deployment uses both subnets.
    • Groups of EC2 instances for each logical role in the Confluent Platform deployment. Instances are distributed evenly between the Availability Zones yet remain a single, logical cluster. The Confluent Platform instances include:
      • Zookeepers, which host the quorum management service and manage topic metadata
      • Brokers, which host the Kafka broker service and maintain topic data
      • Workers, which support the additional Confluent Platform services
    • Security group settings to allow for necessary network traffic into the instances:
      • All Kafka client traffic to zookeepers and brokers
      • REST and HTTP/HTTPS access to additional Confluent Platform services
  •  How to deploy
  • Deploy your Confluent Platform environment on AWS in 10-15 minutes, in a few easy steps:

    1. If you don't already have an AWS account, sign up at https://aws.amazon.com.
    2. Subscribe to a Linux AMI in the AWS Marketplace (Amazon Linux, CentOS 7, or Ubuntu Server 16.04 LTS).
    3. Launch the Quick Start. Each deployment takes about 10 minutes. You can choose from two options:
    4. Test your deployment by running tests from worker nodes.

    To customize your deployment, you can choose the version and edition of Confluent Platform you'd like to deploy; configure the number, type, and storage capacity for zookeeper, broker, and worker instances; and change CIDR block sizes and other configuration settings.   

  •  Cost and licenses
  • You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional cost for using the Quick Start.

    The AWS CloudFormation template for this Quick Start includes configuration parameters that you can customize. Some of these settings, such as instance type, will affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you will be using. Prices are subject to change.

    The Confluent software is provided in a bring-your-own-license model. The Quick Start supports two software editions: Confluent Open Source and Confluent Enterprise. Confluent Open Source does not require a license, while Confluent Enterprise requires the purchase of a license directly from Confluent. For convenience, Confluent Enterprise is deployed with a 30-day trial license by default.