Packaging with Docker

PyMacaron microservices are designed to be packaged as Docker images. pymacaron-docker contains the tools required to build a Docker image containing and starting your microservice.

Prerequisites

You need to be able to run a docker container locally on your development host, as part of the deployment pipeline for PyMacaron microservices. Simply install docker engine as follows:

apt-get install docker docker-engine

And install pymacaron-docker:

pip install pymacaron-docker

Configuration

Make sure that the following key pairs are set in your project’s ‘pym-config.yaml’:

name: <PROJECT NAME>
docker_repo: <MYREPO>

Your microservice’s Docker image will be named ‘-' where '' is a unique number generated based on the date and the git revision number of your project.

Login to your docker repository

Using the same unix user with which you will later run the deployment pipeline, login to the Docker repository to which your microservice’s image will be uploaded (‘docker_repo’ in ‘pym-config.yaml’):

docker login

This creates the file ‘~/.docker/config.json file’ containing your docker auth token.

Manually generating the Docker image

Pymacaron generates a Dockerfile each time a microservice is deployed. The tool ‘pymdocker’ is in charge of generating this Dockerfile, compiling it into a Docker image and uploading that image to your Docker repository, as in the example below:

# Generate a Dockerfile, compile it into a Docker image and upload it to
# your Docker registry.
# pymdocker is normally called within pymdeploy which generates the version
# for each release.
pymdocker --version <VERSION_OF_YOUR_MICROSERVICE>

Type ‘pymdocker –help’ for more usage examples.

Customizing the Dockerfile

You have the possibility to inject extra commands into the Dockerfile generated by ‘pymdocker’. ‘pymdocker’ looks for a file named ‘Dockerfile.extra’ in the root of your project, and if it finds one, its content will be inserted at the end of the Dockerfile, just before the final CMD line.

For example, adding the following Dockerfile.extra in your project will have Chrome installed into your docker image:

$ cat Dockerfile.extra
# Install chrome and chromedriver
RUN apt-get update && apt-get install wget
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add - && \
    sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' && \
    sudo apt-get update && \
sudo apt-get install -y google-chrome-stable
RUN pip install selenium chromedriver-install
RUN python -c "import chromedriver_install as cdi; print(cdi.install(file_directory='./lib/', verbose=True, chmod=True, overwrite=False, vers$

Using a different docker base image

By default, Pymacaron docker images are built on top of the latest https://hub.docker.com/r/pymacaron/basepymacaron/base image. You can change that and use a custom base image by setting the config file parameter ‘docker_base’ in your project’s ‘pym-config.yaml’, as in the example below:

docker_base: pymacaron/scraper:190521-0957-29

Deployment

PyMacaron microservices are designed to support multiple deployment targets. For the moment however, only AWS Beanstalk is supported.

Deployment pipeline with pymdeploy

bin/pymdeploy implements the deployment pipeline of PyMacaron microservices, which consists of the following steps:

1. Execute unittests under ‘test/’ with nosetests. Stop if tests fail.

2. Generate a docker image in which the app starts inside gunicorn.

3. Start this image in a local docker instance and run acceptance tests from the ‘testaccept/’ directory against it. Stop if tests fail.

4. Push that image to hub.docker.com.

5. Deploy the image to a live environment, AWS Beanstalk or other, as specified in pym-config.yaml.

6. Run acceptance tests again, this time against the live environment, as a final validation.

Usage

To execute the full deployment pipeline, do:

cd your-project-root
pymdeploy

For more usage examples, see ‘pymdeploy –help’.