Compose for development stacks

See also

Compose for development stacks

See also

Dockerfiles are great to build container images.

But what if we work with a complex stack made of multiple containers ?

Eventually, we will want to write some custom scripts and automation to build, run, and connect our containers together.

There is a better way: using Docker Compose.

In this section, you will use Compose to bootstrap a development environment

What is Docker Compose ?

See also

Docker Compose (formerly known as fig) is an external tool.

Unlike the Docker Engine, it is written in Python. It’s open source as well.

The general idea of Compose is to enable a very simple, powerful onboarding workflow:

  • Checkout your code.

  • Run docker-compose up.

  • Your app is up and running !

Compose overview

See also

This is how you work with Compose:

  • You describe a set (or stack) of containers in a YAML file called docker-compose.yml.

  • You run docker-compose up.

  • Compose automatically pulls images, builds containers, and starts them.

  • Compose can set up links, volumes, and other Docker options for you.

  • Compose can run the containers in the background, or in the foreground.

  • When containers are running in the foreground, their aggregated output is shown.

Before diving in, let’s see a small example of Compose in action.

Checking if Compose is installed

See also

If you are using the official training virtual machines, Compose has been pre-installed.

You can always check that it is installed by running:

$ docker-compose --version

Launching Our First Stack with Compose

See also

First step: clone the source code for the app we will be working on.

$ cd
$ git clone git://github.com/jpetazzo/trainingwheels
...
$ cd trainingwheels

Second step: start your app.

$ docker-compose up

Watch Compose build and run your app with the correct parameters, including linking the relevant containers together.

$$ cat docker-compose.yml

version: "2"

services:
  www:
        build: www
        ports:
          - 8000:5000
        user: nobody
        environment:
          DEBUG: 1
        command: python counter.py
        volumes:
          - ./www:/src

  redis:
        image: redis

$ tree

.
├── docker-compose.yml
├── docker-compose.yml-ecs
├── ports.yml
└── www
        ├── assets
        │   ├── css
        │   │   ├── bootstrap.min.css
        │   │   └── bootstrap-responsive.min.css
        │   └── js
        │       └── bootstrap.min.js
        ├── counter.py
        ├── Dockerfile
        └── templates
                ├── error.html
                └── index.html

5 directories, 10 files

$ docker-compose up

Creating network "trainingwheels_default" with the default driver
Building www
Step 1/8 : FROM python
latest: Pulling from library/python
cc1a78bfd46b: Pull complete
6861473222a6: Pull complete
7e0b9c3b5ae0: Pull complete
3ec98735f56f: Pull complete
9b311b87a021: Pull complete
048165938570: Pull complete
1ca3d78efb22: Pull complete
0f6c8999c3b7: Pull complete
5a85410f5000: Pull complete
Digest: sha256:52a2bd143faf6430b182b56a5fdeb70f26b8ca8fbd40210c3ed8a8ee1eaba343
Status: Downloaded newer image for python:latest
 ---> 29d2f3226daf
Step 2/8 : RUN pip install flask
 ---> Running in 30e9159dd9dc
Collecting flask
  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
Collecting itsdangerous>=0.24 (from flask)
  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
Collecting Jinja2>=2.10 (from flask)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
Collecting click>=5.1 (from flask)
  Downloading https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)
Collecting Werkzeug>=0.14 (from flask)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Building wheels for collected packages: itsdangerous, MarkupSafe
  Running setup.py bdist_wheel for itsdangerous: started
  Running setup.py bdist_wheel for itsdangerous: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/2c/4a/61/5599631c1554768c6290b08c02c72d7317910374ca602ff1e5
  Running setup.py bdist_wheel for MarkupSafe: started
  Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/33/56/20/ebe49a5c612fffe1c5a632146b16596f9e64676768661e4e46
Successfully built itsdangerous MarkupSafe
Installing collected packages: itsdangerous, MarkupSafe, Jinja2, click, Werkzeug, flask
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-1.0.2 itsdangerous-0.24
Removing intermediate container 30e9159dd9dc
 ---> 715be459df83
Step 3/8 : RUN pip install gunicorn
 ---> Running in 27a29e572569
Collecting gunicorn
  Downloading https://files.pythonhosted.org/packages/55/cb/09fe80bddf30be86abfc06ccb1154f97d6c64bb87111de066a5fc9ccb937/gunicorn-19.8.1-py2.py3-none-any.whl (112kB)
Installing collected packages: gunicorn
Successfully installed gunicorn-19.8.1
Removing intermediate container 27a29e572569
 ---> cd78b2130321
Step 4/8 : RUN pip install redis
 ---> Running in 637a8b3cd24b
Collecting redis
  Downloading https://files.pythonhosted.org/packages/3b/f6/7a76333cf0b9251ecf49efff635015171843d9b977e4ffcf59f9c4428052/redis-2.10.6-py2.py3-none-any.whl (64kB)
Installing collected packages: redis
Successfully installed redis-2.10.6
Removing intermediate container 637a8b3cd24b
 ---> 08766036473f
Step 5/8 : COPY . /src
 ---> 4de5b2a959d5
Step 6/8 : WORKDIR /src
Removing intermediate container 6013def61017
 ---> 54eb5e672592
Step 7/8 : CMD gunicorn --bind 0.0.0.0:5000 --workers 10 counter:app
 ---> Running in bab6ea1f334c
Removing intermediate container bab6ea1f334c
 ---> 585a2f6a0163
Step 8/8 : EXPOSE 5000
 ---> Running in 228ff16daa14
Removing intermediate container 228ff16daa14
 ---> d0ad402a2cc3
Successfully built d0ad402a2cc3
Successfully tagged trainingwheels_www:latest
WARNING: Image for service www was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling redis (redis:)...
latest: Pulling from library/redis
4d0d76e05f3c: Pull complete
cfbf30a55ec9: Pull complete
82648e31640d: Pull complete
fb7ace35d550: Pull complete
497bf119bebf: Pull complete
89340f6074da: Pull complete
Digest: sha256:4aed8ea5a5fc4cf05c8d5341b4ae4a4f7c0f9301082a74f6f9a5f321140e0cd3
Status: Downloaded newer image for redis:latest
Creating trainingwheels_www_1   ... done
Creating trainingwheels_redis_1 ... done
Attaching to trainingwheels_redis_1, trainingwheels_www_1
redis_1  | 1:C 01 Jun 07:45:02.780 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1  | 1:C 01 Jun 07:45:02.780 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1  | 1:C 01 Jun 07:45:02.780 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1  | 1:M 01 Jun 07:45:02.782 * Running mode=standalone, port=6379.
redis_1  | 1:M 01 Jun 07:45:02.782 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1  | 1:M 01 Jun 07:45:02.782 # Server initialized
redis_1  | 1:M 01 Jun 07:45:02.782 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then
reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1  | 1:M 01 Jun 07:45:02.782 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the comma
nd 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1  | 1:M 01 Jun 07:45:02.782 * Ready to accept connections
www_1    |  * Serving Flask app "counter" (lazy loading)
www_1    |  * Environment: production
www_1    |    WARNING: Do not use the development server in a production environment.
www_1    |    Use a production WSGI server instead.
www_1    |  * Debug mode: on
www_1    |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
www_1    |  * Restarting with stat
www_1    |  * Debugger is active!
www_1    |  * Debugger PIN: 313-495-332
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET / HTTP/1.1" 200 -
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET /assets/css/bootstrap.min.css HTTP/1.1" 200 -
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET /assets/css/bootstrap-responsive.min.css HTTP/1.1" 200 -
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET /assets/js/bootstrap.min.js HTTP/1.1" 200 -
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET /favicon.ico HTTP/1.1" 404 -
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET /favicon.ico HTTP/1.1" 404 -
www_1    | X.X.X.X - - [01/Jun/2018 07:49:36] "GET /favicon.ico HTTP/1.1" 404 -

Stopping the app

See also

When you hit ^C, Compose tries to gracefully terminate all of the containers.

After ten seconds (or if you press ^C again) it will forcibly kill them.

^CGracefully stopping... (press Ctrl+C again to force)
Stopping trainingwheels_www_1   ... done
Stopping trainingwheels_redis_1 ... done

The docker-compose.yml file

See also

Here is the file used in the demo:

version: "2"
services:
  www:
        build: www
        ports:
          - 8000:5000
        user: nobody
        environment:
          DEBUG: 1
        command: python counter.py
        volumes:
          - ./www:/src
  redis:
        image: redis

$ cat www/Dockerfile

FROM python
RUN pip install flask
RUN pip install gunicorn
RUN pip install redis
COPY . /src
WORKDIR /src
CMD gunicorn --bind 0.0.0.0:5000 --workers 10 counter:app
EXPOSE 5000

Compose file versions

See also

Version 1 directly has the various containers (www, redis…) at the top level of the file.

Version 2 has multiple sections:

  • version is mandatory and should be “2”.

  • services is mandatory and corresponds to the content of the version 1 format.

  • networks is optional and indicates to which networks containers should be connected. (By default, containers will be connected on a private, per-app network.)

  • volumes is optional and can define volumes to be used and/or shared by the containers.

Version 3 adds support for deployment options (scaling, rolling updates, etc.)

Containers in docker-compose.yml

See also

Each service in the YAML file must contain either build, or image.

  • build indicates a path containing a Dockerfile.

  • image indicates an image name (local, or on a registry).

  • If both are specified, an image will be built from the build directory and named image.

The other parameters are optional.

They encode the parameters that you would typically add to docker run.

Sometimes they have several minor improvements.

Container parameters

See also

  • command indicates what to run (like CMD in a Dockerfile).

  • ports translates to one (or multiple) -p options to map ports. You can specify local ports (i.e. x:y to expose public port x).

  • volumes translates to one (or multiple) -v options. You can use relative paths here.

For the full list, check: https://docs.docker.com/compose/compose-file/

Compose commands

See also

We already saw docker-compose up, but another one is docker-compose build

It will execute docker build for all containers mentioning a build path.

It can also be invoked automatically when starting the application:

docker-compose up --build

Another common option is to start containers in the background:

docker-compose up -d

Check container status

See also

It can be tedious to check the status of your containers with docker ps, especially when running multiple apps at the same time.

Compose makes it easier; with docker-compose ps you will see only the status of the containers of the current stack:

$ docker-compose ps

                 Name                       Command               State           Ports
----------------------------------------------------------------------------------------
trainingwheels_redis_1   docker-entrypoint.sh redis ...   Up      6379/tcp
trainingwheels_www_1     python counter.py                Up      0.0.0.0:8000->5000/tcp

Cleaning up (1)

See also

If you have started your application in the background with Compose and want to stop it easily, you can use the kill command:

$ docker-compose kill

Likewise, docker-compose rm will let you remove containers (after confirmation):

$ docker-compose rm

Going to remove trainingwheels_redis_1, trainingwheels_www_1
Are you sure? [yN] y
Removing trainingwheels_redis_1...
Removing trainingwheels_www_1...

Cleaning up (2)

See also

Alternatively, docker-compose down will stop and remove containers.

It will also remove other resources, like networks that were created for the application.

$ docker-compose down

Stopping trainingwheels_www_1 ... done
Stopping trainingwheels_redis_1 ... done
Removing trainingwheels_www_1 ... done
Removing trainingwheels_redis_1 ... done

Special handling of volumes

See also

Compose is smart. If your container uses volumes, when you restart your application, Compose will create a new container, but carefully re-use the volumes it was using previously.

This makes it easy to upgrade a stateful service, by pulling its new image and just restarting your stack with Compose.

Compose project name

See also

  • When you run a Compose command, Compose infers the “project name” of your app.

  • By default, the “project name” is the name of the current directory.

  • For instance, if you are in /home/zelda/src/ocarina, the project name is ocarina.

  • All resources created by Compose are tagged with this project name.

  • The project name also appears as a prefix of the names of the resources.

  • E.g. in the previous example, service www will create a container ocarina_www_1.

  • The project name can be overridden with docker-compose -p.

Running two copies of the same app

See also

If you want to run two copies of the same app simultaneously, all you have to do is to make sure that each copy has a different project name.

You can:

  • copy your code in a directory with a different name

  • start each copy with docker-compose -p myprojname up

Each copy will run in a different network, totally isolated from the other.

This is ideal to debug regressions, do side-by-side comparisons, etc.