Docker Compose: From Local to Amazon ECS

By using cloud platforms, we can take advantage of different resource configurations and compute capacities. However, deploying containerized applications on cloud platforms is proving to be quite challenging, especially for new users who have no expertise on how to use that platform. As each platform may provide specific APIs, orchestrating the deployment of a containerized application can become a hassle.

Docker Compose is a very popular tool used to manage containerized applications deployed on Docker hosts. Its popularity is maybe due to the simplicity on how to define an application and its components in a Compose file and the compact commands to manage its deployment.

Since cloud platforms for containers have emerged, being able to deploy a Compose application on them is a most-wanted feature by many developers that use Docker Compose for their local development.

In this blog post, we discuss how to use Docker Compose to deploy containerized applications to Amazon ECS. We aim to show how the transition from deploying to a local Docker environment to deploying to Amazon ECS is effortless, the application being managed in the same way for both environments.

Requirements

In order to exercise the examples in this blogpost, the following tools need to be installed locally:

For deploying a Compose file to Amazon ECS, we rely on the new Docker Compose implementation embedded into the Docker CLI binary. Therefore, we are going to run docker compose commands instead of docker-compose. For local deployments, both implementations of Docker Compose should work. If you find a missing feature that you use, report it on the issue tracker.

Throughout this blogpost, we discuss how to:

  1. Build and ship a Compose Application. We exercise how to run an application defined in a Compose file locally and how to build and ship its images to Docker Hub to make them accessible from anywhere.
  2. Create an ECS context to target Amazon ECS.   
  3. Run the Compose application on Amazon ECS. 
  1. Build and Ship a Compose application

Let us take an example application with the following structure:

$ tree myproject/
myproject/
├── backend
│   ├── Dockerfile
│   ├── main.py
│   └── requirements.txt
├── compose.yaml
└── frontend
    ├── Dockerfile
    └── nginx.conf

2 directories, 6 files

The content of the files can be found here. The Compose file define only 2 services as follows:

$ cat compose.yaml
services:
frontend:
  build: frontend
  ports:
    - 80:80
  depends_on:
    - backend
backend:
  build: backend

Deploying this file locally on a Docker engine is quite straightforward:

$ docker compose up -d
[+] Running 3/3
⠿ Network "myproject_default"     Created                     0.5s
⠿ Container myproject_backend_1   Started                     0.7s
⠿ Container myproject_frontend_1  Started                     1.4s

Check the application is running locally:

$ docker ps
CONTAINER ID   IMAGE                COMMAND                    CREATED         STATUS         PORTS                NAMES
eec2dd88fd67   myproject_frontend   "/docker-entrypoint...."   4 seconds ago   Up 3 seconds   0.0.0.0:80->80/tcp   myproject_frontend_1
2c64e62b933b   myproject_backend    "python3 /app/main.py"     4 seconds ago   Up 3 seconds                        myproject_backend_1

Query the frontend:

$ curl localhost:80

          ##         .
    ## ## ##        ==
## ## ## ## ##    ===
/"""""""""""""""""\___/ ===
{                       /  ===-
\______ O           __/
\    \         __/
  \____\_______/

Hello from Docker!

To remove the application:

$ docker compose down
[+] Running 3/3
⠿ Container myproject_frontend_1  Removed                                                 0.5s
⠿ Container myproject_backend_1   Removed                                                10.3s
⠿ Network "myproject_default"     Removed                                                 0.4s

In order to deploy this application on ECS, we need to have the images for the application frontend and backend stored in a public image registry such as Docker Hub. This enables the images to be pulled from anywhere.

To upload the images to Docker Hub, we can set the image names in the compose file as follows:

$ cat compose.yamlservices:
frontend:
  image: myhubuser/starter-front
  build: frontend
  ports:
    - 80:80
  depends_on:
    - backend
backend:
  image: myhubuser/starter-back
  build: backend

Build the images with Docker Compose:

$ docker compose build
[+] Building 1.2s (16/16) FINISHED                                                                                                                                 
=> [myhubuser/starter-front internal] load build definition from Dockerfile            0.0s
=> => transferring dockerfile: 31B                                                     0.0s
=> [myhubuser/starter-back internal] load build definition from Dockerfile 0.0s
...

In the build output we can notice the image has been named and tagged according to the image field from the Compose file.

Before pushing the images to Docker Hub, check to be logged in:

$ docker login
...
Login Succeeded

Push the images:

$ docker compose push
[+] Running 0/16
⠧ Pushing Pushing frontend: f009a503aca1 Pushing              [===========================================...                                        2.7s
...

The images should be stored now in Docker Hub.

  1. Create an ECS Docker Context

To make Docker Compose target the Amazon ECS platform, we need first to create a Docker context of the ECS type. A docker context is a mechanism that allows redirecting commands to different Docker hosts or cloud platforms. 

We assume at this point that we have AWS credentials set up in the local environment for authenticating with the ECS platform. 

To create an ECS context run the following command:

$ docker context create ecs myecscontext
? Create a Docker context using:  [Use arrows to move, type to filter]
  An existing AWS profile
  AWS secret and token credentials
> AWS environment variables

Based on the familiarity with the AWS credentials setup and the AWS tools use, we are prompted to choose between 3 context setups. To skip  the details of  AWS credential setup, we choose the option of using environment variables.

$ docker context create ecs myecscontext
? Create a Docker context using: AWS environment variables
Successfully created ecs context "myecscontext"

This requires to have the AWS_ACCESS_KEY and AWS_SECRET_KEY set in the local environment when running Docker commands that target Amazon ECS.

The current context in use is marked by  * in the output of context listing:

$ docker context lsNAME                TYPE      DESCRIPTION                                   DOCKER ENDPOINT       default *           moby      Current DOCKER_HOST based configuration       unix:///var/run/docker.sockmyecscontext        ecs       credentials read from environment                                                           

To make all subsequent commands target Amazon ECS, make the newly created ECS context the one in use by running:

$ docker context use myecscontext
myecscontext


$ docker context ls
NAME                TYPE         DESCRIPTION                               DOCKER ENDPOINT              
default             moby         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
myecscontext *      ecs          credentials read from environment                                                             
  1.  Run the Compose application on Amazon ECS

An alternative to having it as the context in use is to set the context flag for all commands targeting ECS.

WARNING: Check in advance the cost that the ECS deployment may incur for 2 ECS services, load balancing (ALB), cloud map (DNS resolution) etc. 

For the following commands, we keep ECS context as the current context in use. Before running commands on ECS, make sure the Amazon account credentials grant access to manage resources for the application as detailed in the documentation.

 We can now run a command to check we can successfully access ECS.

$ AWS_ACCESS_KEY="*****" AWS_SECRET_KEY="******" docker compose ls
NAME                                STATUS               

Export the AWS credentials to avoid setting them for every command.

$ export AWS_ACCESS_KEY="*****"
$ export AWS_SECRET_KEY="******"

The deploy the sample application to ECS, we can run the same command as in the local deployment:

$ docker compose up
WARNING services.build: unsupported attribute       
WARNING services.build: unsupported attribute       
[+] Running 18/18
⠿ myproject                      CreateComplete                                     206.0s
⠿ FrontendTCP80TargetGroup       CreateComplete                                       0.0s
⠿ CloudMap                       CreateComplete                                      46.0s
⠿ FrontendTaskExecutionRole      CreateComplete                                      19.0s
⠿ Cluster                        CreateComplete                                       5.0s
⠿ DefaultNetwork                 CreateComplete                                       5.0s
⠿ BackendTaskExecutionRole       CreateComplete                                      19.0s
⠿ LogGroup                       CreateComplete                                       1.0s
⠿ LoadBalancer                   CreateComplete                                     122.0s
⠿ Default80Ingress               CreateComplete                                       1.0s
⠿ DefaultNetworkIngress          CreateComplete                                       0.0s
⠿ BackendTaskDefinition          CreateComplete                                       2.0s
⠿ FrontendTaskDefinition         CreateComplete                                       3.0s
⠿ FrontendServiceDiscoveryEntry  CreateComplete                                       1.0s
⠿ BackendServiceDiscoveryEntry   CreateComplete                                       2.0s
⠿ BackendService                 CreateComplete                                      65.0s
⠿ FrontendTCP80Listener          CreateComplete                                       3.0s
⠿ FrontendService                CreateComplete                                      66.0s

Docker Compose converts the Compose file to a CloudFormation template defining a set of AWS resources. Details on the resource mapping can be found in the documentation. To review the CloudFormation template generated, we can run the command:

$ docker compose convert
WARNING services.build: unsupported attribute       
WARNING services.build: unsupported attribute       
AWSTemplateFormatVersion: 2010-09-09
Resources:
  BackendService:
    Properties:
      Cluster:
        Fn::GetAtt:
        - Cluster
        - Arn
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100...

To check the state of the services, we can run the command:

$ docker compose ps
NAME                                              SERVICE             STATUS              PORTS
task/myproject/8c142dea1282499c83050b4d3e689566   backend             Running            
task/myproject/a608f6df616e4345b92a3d596991652d   frontend            Running             mypro-LoadB-1ROWIHLNOG5RZ-1172432386.eu-west-3.elb.amazonaws.com:80->80/http

Similarly to the local run, we can query the frontend of the application:

$ curl mypro-LoadB-1ROWIHLNOG5RZ-1172432386.eu-west-3.elb.amazonaws.com:80

          ##         .
    ## ## ##        ==
## ## ## ## ##    ===
/"""""""""""""""""\___/ ===
{                       /  ===-
\______ O           __/
\    \         __/
  \____\_______/

Hello from Docker!

We can retrieve logs from the ECS containers by running the compose logs command:

$ docker compose logs
backend  |  * Serving Flask app "main" (lazy loading)
backend  |  * Environment: production
backend  |    WARNING: This is a development server. Do not use it in a production deployment.
backend  |    Use a production WSGI server instead.
...
frontend  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
frontend  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
frontend  | /docker-entrypoint.sh: Configuration complete; ready for start up
frontend  | 172.31.22.98 - - [02/Mar/2021:08:35:27 +0000] "GET / HTTP/1.1" 200 212 "-" "ELB-HealthChecker/2.0" "-"
backend   | 172.31.0.11 - - [02/Mar/2021 08:35:27] "GET / HTTP/1.0" 200 -
backend   | 172.31.0.11 - - [02/Mar/2021 08:35:57] "GET / HTTP/1.0" 200 -
frontend  | 172.31.22.98 - - [02/Mar/2021:08:35:57 +0000] "GET / HTTP/1.1" 200 212 "-" "curl/7.75.0" "94.239.119.152"
frontend  | 172.31.22.98 - - [02/Mar/2021:08:35:57 +0000] "GET / HTTP/1.1" 200 212 "-" "ELB-HealthChecker/2.0" "-"

To terminate the Compose application and release AWS resources, run:

$ docker compose down
[+] Running 2/4
⠴ myproject              DeleteInProgress User Initiated                                        8.5s
⠿ DefaultNetworkIngress  DeleteComplete                                                         1.0s
⠿ Default80Ingress       DeleteComplete                                                         1.0s
⠴ FrontendService        DeleteInProgress                                                       7.5s...

The Docker documentation provides several examples of Compose files, supported features, details on how to deploy and how to update a Compose application running in ECS, etc.

The following features are discussed in detail:

Summary

We have covered the transition from local deployment of a Compose application to the deployment on Amazon ECS. We have used a minimal generic example for demonstrating how to use the Docker Compose cloud-capability. For a better understanding on how to update the Compose file and use specific AWS features, the documentation provides much more details.

Resources: