AWS Partner Network (APN) Blog

Announcing Mesosphere DCOS on AWS

Mesosphere is announcing the general availability of its Datacenter Operating System (DCOS) cluster management product on the Amazon Web Services (AWS) cloud. Let’s take a look at Mesosphere DCOS and launch a simple DCOS cluster on AWS.

What is Mesosphere DCOS?

The Mesosphere Datacenter Operating System (DCOS) is a cluster management platform designed to abstract physical and cloud-based resources behind a single API. This platform enables customers to improve efficiency by running existing applications and complex services like Apache Hadoop, Apache Spark, Apache Kafka, and Apache Hadoop YARN side by side on the same set of servers.

DCOS includes the Apache Mesos, schedulers Chronos and Marathon, an integrated command line interface (CLI) and GUI for administrative tasks, and a simple installer. Once deployed, DCOS enables single-command installation of services like Hadoop or Spark from the DCOS public repository. You can also install custom applications that use language-specific packaging (e.g., a zip archive) or Docker-based container packaging. DCOS is particularly easy to try on AWS: you can launch an entire cloud-based datacenter cluster in minutes by following the steps in the Mesosphere installation guide.

To read about the DCOS architecture, see the overview on the Mesosphere website.

DCOS on AWS

Mesosphere DCOS and AWS are a natural fit: DCOS eases the administrative tasks inherent in managing fleets of containers or complex cluster-enabled services, but it still relies on properly managed compute and networking building blocks like those provided by AWS. By extending on-premises clusters or building entirely new installations on Amazon Elastic Compute Cloud (Amazon EC2), you can take advantage of regional diversity for high availability (HA), robust networking described as code, Auto Scaling, elastic compute capacity, and other core features of the AWS platform. Applications deployed on DCOS in the AWS cloud also have direct access to the wealth of AWS value-added features, including Amazon Simple Storage Service (Amazon S3) for scaled object storage and distribution, Amazon Route 53 for global DNS services, and Amazon CloudFront for edge caching.

Installation Example: Creating an AWS DCOS Cluster Using the AWS CLI

DCOS can be installed by launching a Mesosphere-provided AWS CloudFormation template into your AWS account. AWS CloudFormation provides an easy way to create and manage a collection of AWS resources by declaratively describing the complete environment as code, and maintaining the state of the resources over the complete lifecycle.

In this demonstration, we will be following the procedure from the DCOS installation guide to install a simple, single-server DCOS cluster in the AWS US West (Oregon) region. However, instead of using the AWS EC2 console demonstrated in the guide, we will use the AWS command line interface (AWS CLI) to launch the DCOS stack. We recommend that you review the installation guide to understand the overall installation flow and for up-to-date guidance, tips, and hints.

While the Amazon EC2 console is fully featured and easy to use, automating the installation with the AWS CLI enables you to rapidly iterate and test various deployment parameters, and ultimately improves the quality, repeatability, and maintainability of the installation process through its usage of code.

Prerequisites

Once you’ve installed these components, you can create an AWS DCOS cluster by using AWS CLI commands, as described in the following sections.

Create an Amazon EC2 Key Pair 

Mesosphere DCOS requires an AWS key pair to be specified as part of the cluster installation. From the command line, type:


$ aws --region us-west-2 ec2 create-key-pair --key-name dcos-demo-key --output 
     text \
     --query KeyMaterial > dcos-demo-key.pem
$ chmod 600 dcos-demo-key.pem

Note: Here, we used the aws ec2 create-key-pair command to build a new Amazon EC2 key pair in the US West (Oregon) region. We provided the key name dcos-demo-key and redirected the resulting secret key material into the local file dcos-demo-key.pem. We then protected the key by setting more restrictive permissions on the key file with the chmod command.

Create a Mesosphere DCOS Cluster by Using AWS CloudFormation

Get started with Mesosphere DCOS on AWS by launching an AWS CloudFormation template in the same region as the key pair you created in the previous step. Review the documentation for the location of an appropriate template to use for the value of the template URL. Then, from the command line, type:

 

$ aws --region us-west-2 cloudformation create-stack --stack-name dcos-demo \
--template-url ${TEMPLATE_URL} \
--parameters ParameterKey=AcceptEULA,ParameterValue="Yes" 
     ParameterKey=KeyName,ParameterValue="decos-demo-key" \
--capabilities CAPABILITY_IAM

 

Note: Here, we used the aws cloudformation create-stack command to launch an AWS CloudFormation stack in the US West (Oregon) region. We provided a unique stack name, dcos-demo, a parameter indicating that we accept the EULA, a parameter specifying dcos-demo-key as the name of the key pair we wish to use, and a capability indicating that we are allowing AWS Identity and Access Management (IAM) modifications as a product of the stack launch. Some of the parameters may change depending on the template being used.

Note: In this example, the AWS CloudFormation template we specified with the template-url parameter was replaced with the placeholder ${TERMPLATE_URL}, as the actual templates are region-specific and may be different for your location. The template may also need to be updated as the DCOS stack evolves. Follow the links in the DCOS installation guide for up-to-date template locations and individualized templates for each region.

Tip: The AWS CLI is completely programmable. You can set parameters on the command line (as shown), via a separate JSON file, or via a user-provided URL for full programmatic control.

Monitor the Stack Launch

After 10 to 15 minutes, the stack status should change from CREATE_IN_PROGRESS to CREATE_COMPLETE as in the following example:

During launch:


$ aws --region us-west-2 cloudformation describe-stacks --stack-name dcos-demo
     --query Stacks[0].StackStatus

CREATE_IN_PROGRESS

After launch:


$ aws --region us-west-2 cloudformation describe-stacks --stack-name dcos-demo
     --query Stacks[0].StackStatus

CREATE_COMPLETE

Note: Here, we used the aws cloudformation describe-stacks command to monitor the state of the stack launch. We provided the stack name dcos-demo as before, and used the special query option Stacks[0].StackStatus to limit the output to only the information we are interested in.

Troubleshooting: A ROLLBACK_COMPLETE status means that the deployment has failed. See the AWS CloudFormation console Events tab for useful information about failures. You can also retry the cluster creation. Otherwise, contact Mesosphere support via the feedback mechanisms.

Access the Mesosphere DCOS Dashboard

Once the stack has launched and shows the status CREATE_COMPLETE, run the following command to determine the URL for the DCOS dashboard:

 

$ aws --region us-west-2 cloudformation describe-stacks --stack-name dcos-demo 
     --output json --output text --query Stacks[0].Outputs
Mesos Master DnsAddress dcos-demo-ElasticL-128WE6TIQZZE4-1645934438.us-west- 2.elb.amazonaws.com
Public slaves PublicSlaveDnsAddress     dcos-demo-PublicSl-1L2RTMDHJ3BXH-
     653008986.us-west-2.elb.amazonaws.com

 

Note: We used the aws cloudformation describe-stacks command as in the previous step, but modified the query option to show the output parameters populated after a successful launch.

Enter the value shown for DnsAddress (in bold in the previous example) into a web browser to open the DCOS dashboard. Save this hostname for later use in the DCOS CLI setup process.

If all goes well, you should see a screen similar to the following:

On first launch, the DCOS dashboard will prompt for an email address to use in the Mesosphere DCOS support system.

Install the Mesosphere DCOS CLI

Full DCOS CLI installation instructions are on the Mesosphere DCOS website. To see an abbreviated version of the instructions, choose the Install the Command Line icon in the lower left corner of the DCOS dashboard:

Follow the DCOS CLI installation instructions before proceeding to the next step.

Launch a Sample Application

After you follow the CLI setup instructions, you can start launching applications; for example:

# dcos package install cassandra

Follow the Apache Cassandra installation process on the DCOS dashboard. The Service will appear in response to your CLI command, then show as Unhealthy while the nodes register with one another, and finally after 5-6 minutes will complete installation and show Healthy:

Conclusion

In about 15 minutes, we installed a five-node Marathon-powered Mesos cluster using AWS CLI commands, and then installed Cassandra with a single DCOS CLI command. The cluster is ready for use: you can scale compute capacity by taking advantage of Amazon EC2 Auto Scaling, extend an on-premises DCOS installation, deploy a fully functional web application, and add your own container-based services. In summary, Mesosphere DCOS on AWS represents a very easy, scalable, and cost-effective path for customers and APN Partners who are interested in exploring the wider Mesos ecosystem.