Containers
Enabling continuous workflows for AWS App Runner service with persistency using AWS Copilot CLI
We recently launched a new service called AWS App Runner, the simplest way to build and run your containerized stateless web application on AWS. App Runner provisions and manages all the required resources for you to run containers such as build pipelines, load balancers, scaling in and out, and of course, its underlying infrastructure.
While App Runner works as a great abstraction layer for the web tier and gives you the easiest way to deploy and run stateless web applications, they may require external dependencies. For instance, your web application may need other AWS resources for the data tier such as DynamoDB tables and S3 buckets.
What are the options for you to provision and manage those required resources? Creating them using the management console or the AWS CLI? This would work for some users, but you may want to have a way that is better than it for consistent deployments. How about writing CloudFormation or Terraform templates from scratch? Although this will give you high degree of configurability over the resources with a continuous way to manage them, obviously this wouldn’t be an easy job. We’ve heard from some of the App Runner beta users, that they’re seeking ways to manage dependent resources that are as simple as App Runner.
This is where Copilot comes in. Copilot is an open source command line interface created by AWS, and was originally created to make it easy for developers to build, release, and operate production ready containerized workloads on Amazon ECS and AWS Fargate. Copilot also allows you to create external dependencies such as persistent storage for your application with just few commands. Copilot uses CloudFormation behind the scenes, and it takes care of those dependencies for you. The only thing you need to care about is the “architecture” of your containerized application, instead of thinking and managing underlying infrastructures.
As I mentioned earlier, App Runner gives you fully managed build pipelines. It provides seamless “code-to-deploy” workflow for Node.js and Python applications today, but for some users, it’s important to have their own release pipeline instead of the managed pipeline for achieving more granular controls over it, to run unit/integration tests before deploying the containers for instance. Copilot also helps you with this. It allows you to have your own customizable pipelines with just a few commands.
We launched Copilot v1.7.0 with support for App Runner at the same time as the App Runner GA launch. Now you can run your containerized application on App Runner with Copilot, from small PoC apps to multi-region and multi-account workloads in production.
Let’s dive deep into how they work well together. At the end of the following tutorial, you’ll get an architecture like in the diagram below.
In step 1 and 2, we’re going to set up and deploy minimal required resources such as App Runner and DynamoDB to run the sample application. After that, we’ll have a configurable pipeline with AWS CodePipeline in step 3, then try “push-to-deploy” in step 4 to see all elements work together as expected. Beware that the tutorial may incur some charges especially for App Runner usage and DynamoDB table.
Building and deploying Next.js application on App Runner using Copilot CLI
Prerequisites
- Copilot v1.7.1 or later (see the installation guide if needed)
- Docker Desktop (or Docker Engine on Linux environment)
jq- A forked “hello-app-runner-nodejs” GitHub repository
- You will need your own forked repository to try “push-to-deploy” in the step 4 of this tutorial, so ensure you’re using a forked repo, which you’re allowed to push.
0. Clone forked repo
$ git clone https://github.com/<Your GitHub ID>/hello-app-runner-nodejs
$ cd hello-app-runner-nodejs
1. Create Copilot application
If the Copilot binary is missing or is outdated, follow the Copilot doc to install/update.
# Ensure you're in the root directory of the cloned repository
$ ls
CONTRIBUTING.md Dockerfile LICENSE README.md components docs lib node_modules package-lock.json package.json pages public seed styles
# Ensure you have v1.7.1 or later
$ copilot --version
copilot version: v1.7.1
# Create Copilot app
$ copilot init
# NOTE:
## Choose "n" (No) if Copilot asked "Would you like to use one of your existing applications?"
## Then answer the Copilot's questions as follows
What would you like to name your application?: my-app
Which workload type best represents your architecture?: Request-Driven Web Service (App Runner)
What do you want to name this Request-Driven Web Service?: my-svc
Which Dockerfile would you like to use for my-svc?: ./Dockerfile
Application name: my-app
Service name: my-svc
Dockerfile: ./Dockerfile
Ok great, we'll set up a Request-Driven Web Service named my-svc in application my-app listening on port 80.
✔ Created the infrastructure to manage services and jobs under application my-app.
✔ Wrote the manifest for service my-svc at copilot/my-svc/manifest.yml
Your manifest contains configurations like your container size and port (:80).
✔ Created ECR repositories for service my-svc.
All right, you're all set for local development.Provision “test” environment
At this point you’ll be asked if you want to proceed to create “test” environment. Choose “y” (Yes) to proceed.
Note that Copilot first creates an environment to hold shared infrastructure between services in a Copilot application. Although App Runner does not require a VPC or an Amazon Elastic Container Service (Amazon ECS) cluster, Copilot creates these free-of-charge resources by default. This makes it possible to for you have seamless workflows within the same environment across App Runner or Amazon ECS, whichever you chose for the compute.
# NOTE:
## Since this process is a one-time operation for each environment to create all the shared resources,
## it will take several minutes to complete.
Would you like to deploy a test environment?: y
Deploy: Yes
✔ Linking account <Your AWS account ID> and region <Your AWS region> to application my-app.
✔ Proposing infrastructure changes for the my-app-test environment.
- Creating the infrastructure for the my-app-test environment. [create complete] [77.3s]
- An IAM Role for AWS CloudFormation to manage resources [create complete] [19.1s]
# ~ snip ~
✔ Proposing infrastructure changes for stack my-app-test-my-svc
- Creating the infrastructure for stack my-app-test-my-svc [create complete] [343.5s]
- An IAM Role for App Runner to use on your behalf to pull your image from ECR [create complete] [18.6s]
- An IAM role to control permissions for the containers in your service [create complete] [16.3s]
- An App Runner service to run and manage your containers [create complete] [315.2s]
✔ Deployed my-svc, you can access it at https://<random string>.<your AWS region>.awsapprunner.com.Check deployed app in your web browser
Open the endpoint of your App Runner service in your preferred web browser. You can find the endpoint url at the end of the previous command’s output (it’s in the form of https://<random string>.<your AWS region>.awsapprunner.com).
Once you opened it, you’ll see the app failed to load data. This is because there is no DynamoDB table provisioned at this point.
Okay then, let’s set up a DynamoDB table in the next step.
2. Set up DynamoDB table
$ copilot storage init
What type of storage would you like to associate with my-svc?: DynamoDB (NoSQL)
What would you like to name this DynamoDB Table?: Items
What would you like to name the partition key of this DynamoDB?: ItemId
What datatype is this key?: String
Would you like to add a sort key to this table?: N
Only found one workload, defaulting to: my-svc
Storage type: DynamoDB
Storage resource name: Items
Partition key: ItemId
Partition key datatype: String
Sort key? No
✔ Wrote CloudFormation template for DynamoDB Table Items at copilot/my-svc/addons/Items.yml
Recommended follow-up actions:
- Update my-svc's code to leverage the injected environment variable ITEMS_NAME.
For example, in JavaScript you can write `const storageName = process.env.ITEMS_NAME`.
- Run `copilot deploy --name my-svc` to deploy your storage resources.Before proceeding, it’s helpful to understand that the command copilot storage init does not actually provision AWS resources, it only creates configuration files in your local workspace, more specifically ./copilot/my-svc/addons/Items.yml in this case. So you will provision it using the copilot deploy command in the next step.
You may also realized at the end of the terminal output above, that your application will be able to refer the DynamoDB table name via an environment variable named ITEMS_NAME that Copilot will inject into your App Runner service automatically.
Then, let’s provision the dependent resources (a DynamoDB table and IAM roles in this case) and update your service to use the new environment variable “ITEMS_NAME.”
$ copilot deploy --name my-svc
Only found one environment, defaulting to: test
Environment test is already on the latest version v1.4.0, skip upgrade.
[+] Building 58.6s (12/12) FINISHED
# ~ snip ~
✔ Proposing infrastructure changes for stack my-app-test-my-svc
- Updating the infrastructure for stack my-app-test-my-svc [update complete] [351.9s]
- An Addons CloudFormation Stack for your additional AWS resources [create complete] [60.8s]
- An IAM ManagedPolicy for your service to access the Items db [create complete] [15.7s]
- An Amazon DynamoDB table for Items [create complete] [31.9s]
- An IAM role to control permissions for the containers in your service [update complete] [15.6s]
- An App Runner service to run and manage your containers [update complete] [261.7s]
✔ Deployed my-svc, you can access it at https://<random string>.<your AWS region>.awsapprunner.com.Seed initial data into the DynamoDB table
The provisioned DynamoDB table is still empty of course, so let’s seed it by executing the following command.
If you don’t have npm in your workspace, then just execute ./seed/seed.sh instead. Also note that the script requires the jq command to run successfully.
$ npm run seed
Finding the DynamoDB table name ...
Only found one service, defaulting to: my-svc
DynamoDB table name: my-app-test-my-svc-Items
Seeding initial data to the DynamoDB table ...
Done!Check deployed app in your web browser (again)
Open the endpoint of your App Runner service in your preferred web browser application, or just reload the web page you’ve opened in the previous step. As we did in the previous step, you can obtain the endpoint url from the terminal output (or in the App Runner management console of course).
Congratulations! Now you’re running the app successfully with the loaded data from the DynamoDB table this time ?
Let’s open the first item: “Getting Started with App Runner using Copilot.”
You’ll see that the first item describes what you’ve just finished through this step-by-step guide so far.
We have created several resources by using Copilot, so let’s take a little bit of a closer look into the key component, the App Runner service, in the management console to get a sense of how the App Runner service looks like before going to the next step.
First, open App Runner in the AWS Management Console in your web browser’s new tab and click the service name “my-app-test-my-svc” to open the details. As you can see, it shows the endpoint you’ve accessed several times as “Default domain,” a direct link to the ECR repository, along with multiple tabs such as logs, metrics, and custom domains.
Let’s have a look at the “Logs” tab and you’ll find multiple sections in there. The first one is “Event log.” This shows the latest lifecycle events of your App Runner service. The next one is “Deployment logs,” which shows separated log stream for each deployment. The last one is “Application logs.” These are the actual logs came from your web application running on App Runner. Let’s click and open “Application logs” here.
The logs are stored in CloudWatch Logs, and the console shows the latest combined logs from instances into a single log stream. In the following screenshot, the first instance has generated some error logs that say “Missing required key ‘TableName’ in params.” This is true because we first deployed the application without a DynamoDB table and the application didn’t have the TableName variable at that time. You’ll also find the new instance has started at the line of “Using webpack 5. Reason: …” These are the logs from the instance we’re currently running on App Runner.
As I mentioned at the beginning of this article, Copilot uses CloudFormation to provision and manage AWS resources as its backend. So let’s take a brief look at the CloudFormation stack in the CloudFormation section of the AWS Management Console. Once you opened CloudFormation, select the “my-app-test-my-svc” stack in the “Stacks” list.
The following page can be opened via the “Template” tab, and this is the actual CloudFormation template Copilot generated from ./copilot/my-svc/manifest.yml in your local workspace. It has a definition of the AWS::AppRunner::Service CloudFormation resource, and you can find an environment variable named “ITEMS_NAME” in the resource definition.
The value of the “ITEMS_NAME” environment variable above is referencing the output of the other stack which is the nested stack shown as “my-app-test-my-svc-AddonsStack-O2629IALDI8H” in this case as shown in the following screenshot.
The add-on stack and its CloudFormation template were also created by Copilot from ./copilot/my-svc/addons/Items.yml. It would be also good to take a look at the YAML files to know how Copilot covers and abstracts to build those complex resources with a few questions you answered in the previous steps and the sense of “better by default.”
Okay, let’s get back to the story.
Hope you still have the web page in your web browser with the first item page of the web app. So let’s back to the top page by clicking the “Back to home” link, then open the second item “3. Set-up Release Pipeline” to go to the next step.
3. Set up the release pipeline
As described in the web page you just opened, you’ll create a release pipeline in CodePipeline by using two Copilot commands.
First, you execute copilot pipeline init command to generate configuration files. The files are similar to manifest.yml, which we briefly reviewed in the previous step, but it’s for the pipeline this time.
$ copilot pipeline init
Which environment would you like to add to your pipeline?: test
Which repository would you like to use for your pipeline?: https://github.com//hello-app-runner-nodejs
✔ Wrote the pipeline manifest for hello-app-runner-nodejs at 'copilot/pipeline.yml'
The manifest contains configurations for your CodePipeline resources, such as your pipeline stages and build steps.
Update the file to add additional stages, change the branch to be tracked, or add test commands or manual approval actions.
✔ Wrote the buildspec for the pipeline's build stage at 'copilot/buildspec.yml'
The buildspec contains the commands to build and push your container images to your ECR repositories.
Update the build phase to unit test your services before pushing the images.
Required follow-up actions:
- Commit and push the buildspec.yml, pipeline.yml, and .workspace files of your copilot directory to your repository.
- Run `copilot pipeline update` to create your pipeline.Let’s take a look at the ./copilot/pipeline.yml file.
$ cat ./copilot/pipeline.yml
# This YAML file defines the relationship and deployment ordering of your environments.
# The name of the pipeline
name: pipeline-my-app-hello-app-runner-nodejs
# The version of the schema used in this template
version: 1
# This section defines the source artifacts.
source:
# The name of the provider that is used to store the source artifacts.
provider: GitHub
# Additional properties that further specifies the exact location
# the artifacts should be sourced from. For example, the GitHub provider
# has the following properties: repository, branch.
properties:
branch: main
repository: https://github.com/<Your GitHub ID>/hello-app-runner-nodejs
# Optional: specify the name of an existing CodeStar Connections connection.
# connection_name: a-connection
# The deployment section defines the order the pipeline will deploy
# to your environments.
stages:
- # The name of the environment to deploy to.
name: test
# Optional: flag for manual approval action before deployment.
# requires_approval: true
# Optional: use test commands to validate this stage of your build.
# test_commands: [echo 'running tests', make test]We don’t edit this file in this tutorial, but for example, you can uncomment the test_commands line at the end of the file if you want to add some tests. See the Copilot documentation for the detailed specifications.
Then, execute copilot pipeline update to create a pipeline in CodePipeline in your AWS account.
$ copilot pipeline update
✔ Successfully added pipeline resources to your application: my-app
...Now you need to take manual actions as described in the following terminal output to allow CodePipeline to access your repository.
...
ACTION REQUIRED! Go to https://console.aws.amazon.com/codesuite/settings/connections to update the status of connection copilot-toric-hello-app-runner-n from PENDING to AVAILABLE.
...Open CodeSuite Connections in the console in your web browser’s new tab that is already signed in to your AWS account.
Click the connection name to go to the detailed view.
Click the “Update pending connection” button then you’ll see a popup window titled as “Connect to GitHub.”
If you already have connected GitHub Apps, you’ll see existing GitHub Apps in a dropdown menu once you put the focus into the input next to the magnifier icon.
< I have an existing GitHub App that has access to the repository. >
Select one, and click “Connect” and go to the “Check Copilot’s status” section below.
< I don’t have any existing GitHub App with access to the repository. >
Click the “Install a new app” button then the window loads the GitHub web page to let you select your GitHub namespace that your forked repository belongs to.
Once you selected the GitHub namespace, follow the web page to create or update a GitHub App to allow access to your GitHub repository. Note that you need to choose “hello-app-runner-nodejs” as a selected repository if you choose “Only select repositories” in the “Repository access” section on the GitHub App page.
You’ll be redirected to the AWS Management Console again after clicking the “Save” button in the GitHub page.
Now the input has a numeric value, then click the “Connect” button.
Check Copilot’s status
Go back to the terminal where you executed copilot pipeline update, it must show that you have successfully created a pipeline as follows.
...
✔ Successfully created a new pipeline: pipeline-my-app-hello-app-runner-nodejs
Recommended follow-up actions:
- Run `copilot pipeline status` to see the state of your pipeline.
- Run `copilot pipeline show` for info about your pipeline.Now that you have a release pipeline created in CodePipeline, let’s click the “Back to home” link at the left bottom of the web page in the web application.
In the next step, we’re going to try “push-to-deploy” by using the pipeline we’ve just created.
4. Try “push-to-deploy”
Let’s edit something in the repository and push it to invoke the pipeline!
$ vim ./components/layout.js
# See the `topPageMessage` in the line 10 of the file,
# and change the text "Let's Get Started!" to whatever you'd like to see in the application.
# I choose "Happy building with App Runner and Copilot!" for it :)Once you changed the text, save it and execute the following commands in the terminal window.
# NOTE:
## Ensure you're staging everything under the "copilot" directory,
## not only "./components/layout.js" file.
## The pipeline will fail in its build stage if those files are missing.
$ git add .
$ git commit -m "My first push-to-deploy with CodePipeline"
[main xxxxxxx] My first push-to-deploy with CodePipeline
6 files changed, 265 insertions(+), 1 deletion(-)
create mode 100644 copilot/.workspace
create mode 100644 copilot/buildspec.yml
create mode 100644 copilot/my-svc/addons/Items.yml
create mode 100644 copilot/my-svc/manifest.yml
create mode 100644 copilot/pipeline.yml
$ git push
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 4 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (12/12), 4.52 KiB | 1.51 MiB/s, done.
Total 12 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com//hello-app-runner-nodejs.git
xxxxxxx..yyyyyyy main -> mainAll set! Go to CodePipeline in the console at https://<Your AWS region>.console.aws.amazon.com/codesuite/codepipeline/pipelines in your web browser’s new tab and open the pipeline in the list. You’ll see something like below.
Wait a few minutes for the pipeline to be successfully completed all the stages, then click the “Back to home” link at the left bottom of the web page in the web application. It’ll bring you to the top page.
Congratulations ? As you can see, the change we’ve made were successfully deployed through the pipeline!
Now you have a configurable continuous delivery pipeline that can be invoked by git push. We can push additional commits to see how the sample app works.
5. Cleanup
This is the final (and is optional but important) step. You may want to delete all the provisioned resource to avoid unexpected charges.
With Copilot, it’s also easy to wipe out everything you’ve created by this tutorial in your AWS account. To do this, execute the following command.
$ copilot app delete --name my-app --yes
✔ Deleted service my-svc from environment test.
✔ Deleted resources of service my-svc from application my-app.
✔ Deleted service my-svc from application my-app.
✔ Deleted environment test from application my-app.
✔ Cleaned up deployment resources.
✔ Deleted pipeline pipeline-my-app-hello-app-runner-nodejs from application my-app.
✔ Deleted application resources.
✔ Deleted application configuration.
✔ Deleted local .workspace file.That’s all!
Resources to learn more
There are more resources to learn about AWS App Runner and AWS Copilot CLI, and we encourage you to check out them!
- Introducing AWS App Runner | Containers
- AWS Copilot Documentation
- AWS App Runner Documentation
- AWS App Runner Workshop
- ✨ AWS App Runner Public Roadmap at GitHub ✨ – See the latest roadmap items and leave your requests and feedback to the App Runner team directly!
Happy building with App Runner and Copilot! ?