The docker-compose tool is pretty popular for running dockerized applications in a local development environment. All we need to do is write a Compose file containing the configuration for the application’s services and have a running Docker engine for deployment. From here, we can get the application running locally in a few seconds with a single `docker-compose up` command.
This was the initial scope but…
As developers look to have the same ease-of-deployment in CI pipelines/production environments as in their development environment, we find today docker-compose being used in different ways and beyond its initial scope. In such cases, the challenge is that docker-compose provided support for running on remote docker engines through the use of the DOCKER_HOST environment variable and -H, –host command line option. This is not very user friendly and managing deployments of Compose applications across multiple environments becomes a burden.
To address this issue, we rely on Docker Contexts to securely deploy Compose applications across different environments and manage them effortlessly from our localhost. The goal of this post is to show how to use contexts to target different environments for deployment and easily switch between them.
We’ll start defining a sample application to use throughout this exercise, then we’ll show how to deploy it on the localhost. Further we’ll have a look at a Docker Context and the information it holds to allow us to safely connect to remote Docker engines. Finally, we will exercise the use of Docker Contexts with docker-compose to deploy on remote engines.
Before proceeding, docker and docker-compose must be installed on the localhost. Docker Engine and Compose are included in Docker Desktop for Windows and macOS. For Linux you will need to get Docker Engine and docker-compose. Make sure you get docker-compose with the context support feature. This is available starting with release 1.26.0-rc2 of docker-compose.
Sample Compose application
Let’s define a Compose file describing an application consisting of two services: frontend and backend. The frontend service will run an nginx proxy that will forward the HTTP requests to a simple Go app server.
A sample with all necessary files for this exercise can be downloaded from here or any other sample from the Compose samples repository can be used instead.
The project structure and the Compose file can be found below:
$ tree hello-docker
hello-docker
├── backend
│ ├── Dockerfile
│ └── main.go
├── docker-compose.yml
└── frontend
├── Dockerfile
└── nginx.conf
docker-compose.yml
version: "3.6"
services:
frontend:
build: frontend
ports:
- 8080:80
depends_on:
- backend
backend:
build: backend
Running on localhost
To deploy the application we defined previously, go to the project directory and run docker-compose:
$ cd hello-docker/
$ docker-compose up -d
Creating network "hello-docker_default" with the default driver
Creating hello-docker_backend_1 ... done
Creating hello-docker_frontend_1 ... done
Check all containers are running and port 80 of the frontend service container is mapped to port 8080 of the localhost as described in the docker-compose.yml.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
07b55d101e74 nginx:latest "nginx -g 'daemon of..." 6 seconds ago Up 5 seconds
0.0.0.0:8080->80/tcp hello-docker_frontend_1
48cdf1b8417c hello-docker_backend "/usr/local/bin/back..." 6 seconds ago Up 5 seconds hello-docker_backend_1
Query the web service on port 8080 to get the hello message from the go backend.
$ curl localhost:8080
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
{ / ===-
\______ O __/
\ \ __/
\____\_______/
Hello from Docker!
Running on a remote host
A remote Docker host is a machine, inside or outside our local network which is running a Docker Engine and has ports exposed for querying the Engine API.
The sample application can be deployed on a remote host in several ways. Assume we have SSH access to a remote docker host with a key-based authentication to avoid a password prompt when deploying the application.
There are three ways to deploy it on the remote host:
1. Manual deployment by copying project files, install docker-compose and running it
A common usage of Compose is to copy the project source with the docker-compose.yml, install docker-compose on the target machine where we want to deploy the compose app and finally run it.
$ scp -r hello-docker user@remotehost:/path/to/src
$ ssh user@remotehost
$ pip install docker-compose
$ cd /path/to/src/hello-docker
$ docker-compose up -d
The disadvantages in this case is that for any change in the application sources or Compose file, we have to copy, connect to the remote host and re-run.
2. Using DOCKER_HOST environment variable to set up the target engine
Throughout this exercise we use the DOCKER_HOST environment variable scenario to target docker hosts, but the same can be achieved by passing the -H, –host argument to docker-compose.
$ cd hello-docker
$ DOCKER_HOST=“ssh://user@remotehost” docker-compose up -d
This is a better approach than the manual deployment. But it gets quite annoying as it requires to set/export the remote host endpoint on every application change or host change.
3. Using docker contexts
$ docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
…
remote ssh://user@remotemachine
$ cd hello-docker
$ docker-compose ‐‐context remote up -d
Docker Contexts are an efficient way to automatically switch between different deployment targets. We will discuss contexts in the next section in order to understand how Docker Contexts can be used with compose to ease / speed up deployment.
Docker Contexts
A Docker Context is a mechanism to provide names to Docker API endpoints and store that information for later usage. The Docker Contexts can be easily managed with the Docker CLI as shown in the documentation.
Create and use context to target remote host
To access the remote host in an easier way with the Docker client, we first create a context that will hold the connection path to it.
$ docker context create remote ‐‐docker “host=ssh://user@remotemachine”
remote
Successfully created context “remote”
$ docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default * Current DOCKER_HOST… unix:///var/run/docker.sock swarm
remote ssh://user@remotemachine
Make sure we have set the key-based authentication for SSH-ing to the remote host. Once this is done, we can list containers on the remote host by passing the context name as an argument.
$ docker ‐‐context remote ps
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
We can also set the “remote” context as the default context for our docker commands. This will allow us to run all the docker commands directly on the remote host without passing the context argument on each command.
$ docker context use remote
remote
Current context is now “remote”
$ docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default Current DOCKER_HOST … unix:///var/run/docker.sock swarm
remote * ssh://user@remotemachine
docker-compose context usage
The latest release of docker-compose now supports the use of contexts for accessing Docker API endpoints. This means we can run docker-compose and specify the context “remote” to automatically target the remote host. If no context is specified, docker-compose will use the current context just like the Docker CLI.
$ docker-compose ‐‐context remote up -d
/tmp/_MEI4HXgSK/paramiko/client.py:837: UserWarning: Unknown ssh-ed25519 host key for 10.0.0.52: b’047f5071513cab8c00d7944ef9d5d1fd’
Creating network “hello-docker_default” with the default driver
Creating hello-docker_backend_1 … done
Creating hello-docker_frontend_1 … done
$ docker ‐‐context remote ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
ddbb380635aa hello-docker_frontend “nginx -g ‘daemon of…” 24 seconds ago
Up 23 seconds 0.0.0.0:8080->80/tcp hello-docker_web_1
872c6a55316f hello-docker_backend “/usr/local/bin/back…” 25 seconds ago
Up 24 seconds hello-docker_backend_1
Compose deployments across multiple targets
Many developers may have several development/test environments that they need to switch between. Deployment across all these is now effortless with the use of contexts in docker-compose.
We now try to exercise context switching between several Docker engines. For this, we define three targets:
- Localhost running a local Docker engine
- A remote host accessible through ssh
- A Docker-in-Docker container acting as another remote host
The table below shows the mapping a contexts to docker targets:
| Target Environment | Context name | API endpoint |
| localhost | default | unix:///var/run/docker.sock |
| Remote host | remote | ssh://user@remotemachine |
| docker-in-docker | dind | tcp://127.0.0.1:2375 |
To run a Docker-in-Docker container with the port 2375 mapped to localhost run:
$ docker run ‐‐rm -d -p “2375:2375” ‐‐privileged -e “DOCKER_TLS_CERTDIR=” ‐‐name dind docker:19.03.3-dind
ed92bc991bade2d41cab08b8c070c70b788d8ecf9dffc89e8c6379187aed9cdc
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
ed92bc991bad docker:19.03.3-dind “dockerd-entrypoint.…” 17 seconds ago Up 15 seconds
0.0.0.0:2375->2375/tcp, 2376/tcp dind
Create a new context ‘dind’ to easily target the container:
$ docker context create dind ‐‐docker “host=tcp://127.0.0.1:2375” ‐‐default-stack-orchestrator swarm
dind
Successfully created context “dind”
$ docker context ls
NAME DESCRIPTION
DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default * Current DOCKER_HOST … unix:///var/run/docker.sock swarm
remote ssh://user@devmachine swarm
We can now target any of the environments to deploy the Compose application from the localhost.
$ docker context use dind
dind
Current context is now “dind”
$ docker-compose up -d
Creating network “hello-docker_default” with the default driver
Creating hello-docker_backend_1 … done
Creating hello-docker_frontend_1 … done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
951784341a0d hello-docker_frontend “nginx -g ‘daemon of…” 34 seconds ago
Up 33 seconds 0.0.0.0:8080->80/tcp hello-docker_frontend_1
872c6a55316f hello-docker_backend “/usr/local/bin/back…” 35 seconds ago
Up 33 seconds hello-docker_backend_1
$ docker ‐‐context default ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
ed92bc991bad docker:19.03.3-dind “dockerd-entrypoint….” 28 minutes ago
Up 28 minutes 0.0.0.0:2375->2375/tcp, 2376/tcp dind
$ docker-compose ‐‐context remote up -d
/tmp/_MEIb4sAgX/paramiko/client.py:837: UserWarning: Unknown ssh-ed25519 host key for 10.0.0.52: b’047f5071513cab8c00d7944ef9d5d1fd’
Creating network “hello-docker_default” with the default driver
Creating hello-docker_backend_1 … done
Creating hello-docker_frontend_1 … done
$ docker context use default
default
Current context is now “default”
$ docker-compose up -d
Creating network “hello-docker_default” with the default driver
Creating hello-docker_backend_1 … done
Creating hello-docker_frontend_1 … done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
077b5e5b72e8 hello-docker_frontend “nginx -g ‘daemon of…” About a minute ago
Up about a minute 0.0.0.0:8080->80/tcp hello-docker_frontend_1
fc01878ad14e hello-docker_backend “/usr/local/bin/back…” About a minute ago
Up about a minute hello-docker_backend_1
ed92bc991bad docker:19.03.3-dind “dockerd-entrypoint….” 34 minutes ago
Up 34 minutes 0.0.0.0:2375->2375/tcp, 2376/tcp dind
The sample application runs now on all three hosts. Querying the frontend service on each of these hosts as shown below should return the same message:
$ curl localhost:8080
$ docker exec -it dind sh -c “wget -O – localhost:8080”
$ curl 10.0.0.52:8080
Output:
## .
## ## ## ==
## ## ## ## ## ===
/”””””””””””””””””\___/ ===
{ / ===-
\______ O __/
\ \ __/
\____\_______/
Hello from Docker!
Summary
Deploying to remote hosts with docker-compose has been a common use-case for quite some time.
The Docker Contexts support in docker-compose offers an easy and elegant approach to target different remote hosts. Switching between different environments is now easy to manage and deployment risks across them are reduced. We have shown an example of how to access remote docker hosts via SSH and tcp protocols hoping these cover a large number of use-cases.