Friday, December 9, 2016

Backdoors for becoming root in a Docker container.

In my last post, the main issue I looked at was whether you can trust what a Docker-formatted image says about the user it will run as. What we found was that if the ‘USER’ statement is used in a Dockefile, but is set to a name, you have no idea what UNIX user ID the application in the container will run as. This is because the name could be mapped to any user ID by the UNIX passwd file.

Setting up the UNIX passwd file such that a user name other than ‘root’ also mapped to the UID of 0 provided a backdoor to becoming root in the running container. By requiring that an integer UID be used with the ‘USER’ statement in a Dockerfile, we can inspect the image metadata and decide not to run the image if ‘USER’ wasn’t a non zero integer value.

Is this enough to protect us though? Are there other backdoors for becoming ‘root' in a Docker container. The answer to that is that there is, and this post will look at some of these ways.

Creating a setuid executable

The primary path for switching from a non privileged user to the ‘root’ user on a UNIX system is a setuid executable. This is an executable that has been blessed in such a way that instead of running as the user that ran it, it runs as the user who is the owner of the executable. Such setuid executables will also work inside of a Docker container.

To illustrate how a setuid executable works, lets look at the UNIX utility called ‘id', which is normally used to display information about what user and group the invoking process runs as. If run normally in our Docker container, we might see:

$ id
uid=1001(app) gid=1001(app) groups=1001(app)

Lets create a setuid version of the executable which is owned by ‘root' and bundle that in our image.

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN cp /usr/bin/id /usr/bin/id-setuid-root
RUN chmod 4711 /usr/bin/id-setuid-root
WORKDIR /app
USER 1001

Running the original version of ‘id’ and the setuid version, we get:

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ id-setuid-root
uid=1001(app) gid=1001(app) euid=0(root) groups=1001(app)

As can be seen, the result is that although the real user ID is the same, the effective user ID is that of the ‘root’ user. This means that by using a setuid executable, we gain the rights to run something as if we are ‘root’, or at least very close to being ‘root’. I say very close to being ‘root’ as it is only the effective user ID which is ‘root’ and not the real user ID. In most cases it doesn’t matter, but it hardly matters anyway, as we could also switch our real identify to the ‘root’ user relatively easily from a custom setuid executable of our own.

Running programs as 'root'

In the above example we took an existing executable and made it setuid as the ‘root’ user. We can’t go and do this for every executable we want to run as ‘root’, so what do we do if we want to run an arbitrary executable as ‘root’?

You might think that is simple. All we need to do is make a copy of ‘/bin/bash’ and make it setuid as the ‘root’ user. If we can then run that, we can become ‘root’ and run any program we want as the ‘root’ user.

So this time to create the image we use:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN cp /bin/bash /bin/bash-setuid-root
RUN chmod 4711 /bin/bash-setuid-root
WORKDIR /app
USER 1001

Running our setuid version of bash though, we don’t get what we expected.

bash-4.2$ id
uid=1001(app) gid=1001(app) groups=1001(app)
bash-4.2$ bash-setuid-root
bash-setuid-root-4.2$ id
uid=1001(app) gid=1001(app) groups=1001(app)

This doesn’t work because modern implementations of shells have checks builtin which look for the specific case of where they are executed with an effective user ID of ‘root’, but a non ‘root’ real user ID. In this case, just to make it harder to use this sort of backdoor, they will revert back to running as the real user ID for the effective user ID.

Since this doesn’t work, lets look at how we would achieve the same thing if we weren’t trying to use a backdoor.

The first method we would normally use to execute a command as the ‘root’ user when we are not a privileged user, is to use the ‘sudo’ command. An alternative is to use the ‘su’ command to login as the ‘root’ user.

Because in a Docker image we can install and configure anything we want, there is no reason why we can’t just set these up and use them.

FROM centos:centos7
RUN yum install -y sudo
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
# Allow anyone in group 'app' to use 'sudo' without a password.
RUN echo '%app ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
# Set the password for the 'root' user to be an empty string.
RUN echo 'root:' | chpasswd
WORKDIR /app
USER 1001

With this we can very easily get an interactive shell as the ‘root’ user using ’sudo’, not even requiring a password.

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ sudo -s
# id
uid=0(root) gid=0(root) groups=0(root)

We can also just login as the ‘root’ user, supplying our empty password.

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ su root
Password:
# id
uid=0(root) gid=0(root) groups=0(root)

In both cases the real user ID is that of the ‘root’ user and not just the effective user ID.

So we didn’t even need to fiddle with a backdoor, we can just use the existing features of the operating system. We just need to install the ‘sudo’ package and configure it, or set the ‘root’ password. As it happens, both these mechanisms rely on a setuid executable, but combine it with configuration to guard against who can access them. It is a simple matter though to enable that access given that during the build of a Docker image you can change anything.

You can’t completely block 'root'

You might be thinking at this point that if we can become the ‘root’ user in these ways, what is the point then of using a check on what ‘USER’ specified for the image in the first place. Someone can always set it as a non ‘root’ user, using an integer UID to avoid any restriction on using the image, but then use a custom built backdoor marked as a setuid executable, or using existing system tools such as ‘sudo’, or ‘su’.

What is important to understand is that good security is based on having many layers. You don’t rely on just a single security measure to protect your system. Each extra layer you can add, acts as an obstacle to someone reaching their end goal. Not allowing images to run that don’t set ‘USER’ to a non zero integer ID, would be just one step you can take in a overall security plan.

So it isn’t a waste of time just yet. This is because, although there are ways of becoming the ‘root’ user even if ‘USER’ did not originally declare the container should run as ‘root’, we can still control what the ‘root’ user is actually able to do. This is achieved using Linux capabilities, and is the next layer of defence you should employ.

In the next blog post I will look at Linux capabilities and how to use Docker to restrict what someone could do even if they become the ‘root’ user.

Thursday, December 1, 2016

What USER should you use to run Docker images.

If you follow this blog and my rants on Twitter you will know that I often complain about the prevalence of Docker-formatted container images that will only work if run as the root user, even though there is no technical reason to run them as root. With more and more organisations moving towards containers and using these images in production, some at least are realising that running them as root is probably not a good idea after all. As such, organisations are for their own images at least, starting to create basic guidelines for their developers to follow around what user an image should run as.

A typical example of the most basic guidelines you can find are:

  1. Create a new UNIX group called ‘app’ with a group ID (gid) of 1001.
  2. Create a new UNIX account with user name ‘app’ with a user ID (uid) of 1001, with it being a member of the group ‘app’ and where the home directory of this user is the directory ‘/app’.
  3. Put all your application source code under the ‘/app’ directory.
  4. Set the working directory for any application run to the ‘/app’ directory.
  5. Set the user that the image will run as to the ‘app’ user.

All looks good, and better than running as the root user you might be thinking. Unfortunately there are still a number of problems with these guidelines, as well as things that are missing.

In this blog post I am going to look at the last guideline in that list, and issues around how you specify what user an image should run as. In subsequent posts I will pull apart the other guidelines. At the end of the posts I will summarise what I believe are a better set of basic guidelines around setting up a Docker-formatted container image.

Skeleton for a Dockerfile

Following the above guidelines, the skeleton for the Dockerfile would look like:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
COPY . /app
WORKDIR /app
USER app

If we build and then run this image and have it start up an interactive shell, we can validate that the command we have run is running as the user ‘app’ and that we are in the correct directory.

$ docker run -it --rm best-practices

[app@ca172749cd4f ~]$ id
uid=1001(app) gid=1001(app) groups=1001(app)

[app@ca172749cd4f ~]$ pwd
/app

[app@ca172749cd4f ~]$ ls -las
total 24
4 drwx------ 2 app app 4096 Dec 1 03:15 .
4 drwxr-xr-x 27 root root 4096 Dec 1 03:15 ..
4 -rw-r--r-- 1 app app 18 Aug 2 16:00 .bash_logout
4 -rw-r--r-- 1 app app 193 Aug 2 16:00 .bash_profile
4 -rw-r--r-- 1 app app 231 Aug 2 16:00 .bashrc
4 -rw-r--r-- 1 root root 136 Dec 1 03:15 Dockerfile

[app@ca172749cd4f ~]$ exit

Seems simple enough, so why is this a problem?

Who do you think you can trust?

The problem is the ‘USER’ statement added to the Dockerfile. This is what declares what user the container should run as.

We can see that this was the last statement in the Dockerfile, and so this should be what user is used when the image is run. That this is the case, can be validated by inspecting the meta data of the Docker-formatted image using the ‘docker inspect’ command:

$ docker inspect --format='{{.Config.User}}' best-practices
app

This means that you could verify that an image satisfies the guideline that it runs as the ‘app’ user and not as the root user, before actually running it.

The problem is this doesn’t actually guarantee anything. This is because the value associated with the ‘.Config.User’ setting is a name. You cannot tell what UNIX user ID this really maps to inside of the container when run.

To illustrate the problem, consider the changed Dockerfile as follows:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN sed -i -e 's/1001/0/g' /etc/passwd
COPY . /app
WORKDIR /app
USER app

Validating what ‘docker inspect’ says about what user the image will run as we still get the ‘app’ user:

$ docker inspect --format='{{.Config.User}}' best-practices
app

When we do run the actual image though, that isn’t the case in practice.

$ docker run -it --rm best-practices

[root@71150399f77f app]# id
uid=0(root) gid=0(root) groups=0(root)

So although the Dockerfile specified ‘USER app’ and ‘docker inspect’ also indicated that the image will run as ‘app’, the command actually ran as the root user.

This is because when using a name for ‘USER’, it still needs to be mapped to an actual UNIX user ID by the UNIX passwd file. As shown, we can remap the user name to the user ID ‘0’, meaning it still runs as the root user. We did this surreptitiously to indicate the problem of whether you can actually trust what you see. In this case the command to modify the passwd file was in the Dockerfile and in plain site, but it could also have been buried deep inside some script file or program that had been copied into the Dockerfile and then run during the building of the image.

Why does this matter?

If this is your own system you are running on for your own personal use, then you may not care. It is though a problem in a corporate setting, or if you are running a multi tenant hosting environment where you are allowing Docker-formatted images from potentially untrusted sources. In this case you want to be sure you aren’t going to run an image which actually runs as root. As we have seen, even if ‘USER’ is set in the Dockerfile to be a user other than ‘root’ it doesn’t mean it still isn’t running as root.

Verifying the user is not root

How then can we be confident that a Docker-formatted image we have been supplied isn’t going to run as root? As we have seen we obviously can’t trust ‘USER’ when it is set to a name, we have to reject any such image and not allow it to be run.

The solution is not to use a name, but the actual UNIX user ID with the ‘USER’ statement. What we therefore would require is that the Dockerfile be written as:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN sed -i -e 's/1001/0/g' /etc/passwd
COPY . /app
WORKDIR /app
USER 1001

 Inspect the image now using ‘docker inspect’ and we get:

$ docker inspect --format='{{.Config.User}}' best-practices
1001

Run the image and we get:

$ docker run -it --rm best-practices
bash-4.2$ id
uid=1001 gid=0(root) groups=0(root)
bash-4.2$ ls -las
total 36
4 drwx------ 2 1001 app 4096 Dec 1 05:10 .
4 drwxr-xr-x 28 root root 4096 Dec 1 05:12 ..
4 -rw-r--r-- 1 1001 app 18 Aug 2 16:00 .bash_logout
4 -rw-r--r-- 1 1001 app 193 Aug 2 16:00 .bash_profile
4 -rw-r--r-- 1 1001 app 231 Aug 2 16:00 .bashrc
4 -rw-r--r-- 1 root root 177 Dec 1 05:10 Dockerfile

Thus by requiring that ‘USER’ be set to a UNIX user ID, we are able to guarantee that it will run as the user it says it is. Even if the supplier of the image had still fiddled with the passwd file it wouldn’t matter, they can’t change the fact it will run as that user ID.

Recommended Guidelines

What then is a better guideline about what user a Docker-formatted container should be run? I would suggest the following.

Do not run a Docker-formatted container image as the root user. Always override in the Dockerfile what user the image will run as. This should be done by adding the ‘USER’ statement in the Dockerfile. The value of the ‘USER’ statement must be the integer UNIX user ID of the UNIX account you want any application to run as inside of the container. It should not be the user name for the UNIX account.

In addition to that guideline for the author of any Docker-formatted container image, I would also add the following guideline for anyone building a system on top of the Docker service for running images.

Where it is intended not to allow images to run as the root user, but you want to allow an image to run as the user it specifies, reject any Docker-formatted container image that you can't verify what UNIX user ID it will run as. Use ‘docker inspect’ to determine the user it should run as. Reject the image and do not run it if the user setting specified in the image meta data, is not an integer value greater than 0.

Already you will find some orchestration systems for managing containers using the Docker runtime implement this latter recommendation in certain configurations. One such example is Kubernetes and systems based around it, such as OpenShift. Because of the growth of interest in Kubernetes, especially for enterprise usage and for hosting services, adhering to the first guideline is also the first step in ensuring you will be able to deploy your images to these systems when they are set up in a secure way.

In a followup post I will look at some more aspects around what user an image should run as, whether that be the choice of the developer of the image, or whether it is a user enforced by the hosting service.

Monday, August 1, 2016

Testing out deployment of Python based Opal health care framework.

When I was working on mod_wsgi, but also in a previous job where I was working on web application performance monitoring tools, I was always after good sample Python web applications to test with. Unlike other programming languages for the web there weren’t many end user applications written in Python that you could quickly download and get running. Most of what existed out there were incomplete framework extensions which you still had to customise to get running for your own personal needs. Even if they did provide a way of starting up a skeletal application to at least see what they did, the steps to get them running were often quite complex.

One of the problems with deploying Python web applications you download are that they are often set up to be run in a very specific way with a particular hosting service or WSGI server. This fact meant you could end up spending quite a bit of time fiddling with it to get it all running in the environment you have. This made the exercise of trying to use a Python web application for testing quite frustrating at times. I can quite easily imagine that for users who might be trying to evaluate an Open Source framework extension to see if they could use it, that such difficulties in getting it running could be quite a turn off.

Where I am currently working at Red Hat as part of the OpenShift evangelist team, we are currently running a hackathon (ends 21st September) where the theme is health related applications. I have already seen that there are some Python developers out there participating in the hackathon, so I thought I might do a bit of a search around to see what I could find in the way of existing web applications or framework extensions out there for the Python programming language and test out deploying them using my warpdrive package. I actually wasn’t really expecting to find anything too interesting, but was pleasantly surprised.

One framework extension I found which I thought was quite interesting was called Opal. The Opal framework makes use of Django, along with toolkits for developing the front end such as Angular JS and Bootstrap. It was created by Open Health Care UK. The point of this blog post, as well as highlighting what looks like a quite interesting package out there that you can use, is to see how my warpdrive package stacks up when trying to deploy an arbitrary Python web application off the Internet.

Getting Opal running locally

First up is getting Opal running locally. For this Opal provides some good documentation and also a starter script to get you going. Installation and creation of an initial application I could test with was as simple as running:

pip install opal
opal startproject mynewapp

Once this was done to start up the starter application you run:

cd mynewapp
python manage.py runserver

From a browser you could then visit 'http://localhost:8000' and even login to the admin interface using a pre created user account. The latter was possible as Opal has added hooks which are automatically triggered when ‘runserver’ is used, which will set up the database and create a super user account. They have therefore optimised things for the local developer experience when using the builtin Django development server.

What now though if you wanted to deploy Opal to a production environment? They do provide a ‘Procfile’ for Heroku, but don’t provide anything which really helps out if you want to deploy to another WSGI server such as Apache/mod_wsgi or uWSGI, in a container using a local Docker service, or other PaaS environments such as OpenShift. 

It is making deployment of Python web applications easy across such different environments that my warpdrive project is targeting, so lets now look at using warpdrive to do this.

Preparing the project for warpdrive

With warpdrive already installed, the first thing we want to do is activate a new project using it. In the ‘mynewapp’ directory we run ‘warpdrive project opal’.

$ warpdrive project opal
Initializing warpdrive project 'opal'.
(warpdrive+opal) $

What this command will do is create us a new Python virtual environment just for this application and activate it. This will be an empty Python virtual environment, so next we need to install all the Python packages that the project requires.

When we originally create the project using the ‘opal startproject’ command this conveniently created for us a ‘requirements.txt’ file. This can be used with ‘pip’ to install all the packages, but we aren’t actually going to do that. This is because warpdrive also knows about ‘requirements.txt’ files and we can use it to install the required packages.

Rather than run ‘pip’ directly, we are therefore going to run ‘warpdrive build’ instead. This will not only ensure that any required Python packages are installed, but also ensures that any other framework specific build steps are also run. The output from running ‘warpdrive build’ starts out with:

 -----> Installing dependencies with pip (requirements.txt)
Collecting cryptography==1.3.2 (from -r requirements.txt (line 2))
  Using cached cryptography-1.3.2-cp27-none-macosx_10_6_intel.whl
Collecting Django==1.8.3 (from -r requirements.txt (line 3))
  Using cached Django-1.8.3-py2.py3-none-any.whl
...
Obtaining opal from git+git://github.com/openhealthcare/opal.git@master#egg=opal (from -r requirements.txt (line 18))
  Cloning git://github.com/openhealthcare/opal.git (to master) to /tmp/warpdrive-build.12067/opal
...
Installing collected packages: pycparser, cffi, pyasn1, six, idna, ipaddress, enum34, cryptography, Django, coverage, dj-database-url, gunicorn, psycopg2, static3, dj-static, django-reversion, django-axes, ffs, MarkupSafe, jinja2, letter, requests, djangorestframework, django-appconf, django-compressor, meld3, supervisor, python-dateutil, pytz, billiard, anyjson, amqp, kombu, celery, django-celery, opal
  Running setup.py develop for opal
Successfully installed Django-1.8.3 MarkupSafe-0.23 amqp-1.4.9 anyjson-0.3.3 billiard-3.3.0.23 celery-3.1.19 cffi-1.7.0 coverage-3.6 cryptography-1.3.2 dj-database-url-0.2.1 dj-static-0.0.6 django-appconf-1.0.2 django-axes-1.4.0 django-celery-3.1.17 django-compressor-1.5 django-reversion-1.8.7 djangorestframework-3.2.2 enum34-1.1.6 ffs-0.0.8.1 gunicorn-0.17.4 idna-2.1 ipaddress-1.0.16 jinja2-2.8 kombu-3.0.35 letter-0.4.1 meld3-1.0.2 opal psycopg2-2.5 pyasn1-0.1.9 pycparser-2.14 python-dateutil-2.4.2 pytz-2016.6.1 requests-2.7.0 six-1.10.0 static3-0.7.0 supervisor-3.0
Collecting mod_wsgi
Installing collected packages: mod-wsgi
Successfully installed mod-wsgi-4.5.3

One thing of note here is that ‘pip’ when run is actually trying to install Opal direct from the Git repository on GitHub. This is because the ‘requirements.txt’ file generated by ‘opal startproject’ contains:

-e git://github.com/openhealthcare/opal.git@master#egg=opal

As far as deploying to a production environment, pulling package code direct from a Git repository, and especially from head of the master branch isn’t necessarily the best idea. We instead want to ensure that we are always using a known specific version of the package which we have tested with. To remedy this, this time we will run ‘pip’ directly, but only to uninstall the version of ‘opal’ installed so it will not cause a problems when trying to reinstall it from PyPi.

pip uninstall opal

We now edit the ‘requirements.txt’ file and replace that line with:

opal==0.7.0

Worth highlighting is that this isn’t being done specially because of warpdrive. It is simply good practice to be using pinned versions of packages in a production environment so you know what you are getting. I can only imagine the ‘requirements.txt’ file is generated in this way as it makes the Opal developers job easy when testing themselves when they are working on it.

Having fixed that, we can rerun ‘warpdrive build’ and it will trigger ‘pip’ once more to ensure we have the packages we need installed, and since we removed the ‘opal’ package, it will now install the version we actually want.

Beyond installing any required Python packages, one other thing that warpdrive will do is that it will realise that the Django web framework is being used and will automatically trigger the Django ‘collectstatic’ command to collate together any static files used by the application. The next thing after package installation we therefore see in the output of ‘warpdrive build’ is:

-----> Collecting static files for Django
...
OSError: [Errno 2] No such file or directory: '/Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/static'

Unfortunately this fails though. The reason it fails is actually because the Django settings module for the generated Opal project contains:

# Additional locations of static files
STATICFILES_DIRS = (
os.path.join(PROJECT_PATH, 'static'),
)

With this setting, when ‘collectstatic’ is run, it expects that directory to actually exist and if it doesn’t it will fail.

This is easily fixed by creating the directory:

mkdir mynewapp/static

The directory should though have been created automatically by the ‘opal startproject’ command. That it doesn’t has already been addressed for version 0.7.1 of Opal.

After fixing this, ‘warpdrive build’ then completes successfully. It may have looked a bit messy, but that was only because we had to correct the two things related to the project that ‘opal startproject’ created for us. We could simply have left it using the ‘opal’ project from the Git repository, but felt it better to clarify what is best practice in this case.

Starting up the application

When we first started up the application we used Django’s builtin development server. This server should not though be used in a production system. Instead of the development server you should use a production grade WSGI server such as Apache/mod_wsgi, gunicorn, uWSGI or Waitress. For most use cases any of these WSGI servers will be suitable, but depending on your specific requirements you may find one more appropriate.

Setting up a project for one WSGI server even can still be a challenge in itself for many people. Trying to set up a project for more than one WSGI server so you can compare them only replicates the pain. Usually people will totally muck up the configuration of one or the other and get a totally incorrect impression of which one may actually be better.

In addition to aiming to simplify the build process, another aim of warpdrive is therefore to make it much easier to run up your WSGI application, no matter what WSGI server you want to use. This means you can get started much more quickly, but also give you the flexibility to swap between different WSGI servers.

That said, having prepared our application for warpdrive, to actually run it up and have it start accepting web requests, all we now need to do is run ‘warpdrive start’.

  -----> Configuring for deployment mode: of 'auto'
  -----> Default WSGI server type is 'mod_wsgi'
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
  -----> Running server script start-mod_wsgi
  -----> Executing server command 'mod_wsgi-express start-server --log-to-terminal --startup-log --port 8080 --application-type module --entry-point mynewapp.wsgi --callable-object application --url-alias /assets/ /Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/assets/'
Server URL : http://localhost:8080/
Server Root : /tmp/mod_wsgi-localhost:8080:502
Server Conf : /tmp/mod_wsgi-localhost:8080:502/httpd.conf
Error Log File : /dev/stderr (warn)
Startup Log File : /dev/stderr
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : en_AU.UTF-8
[Mon Aug 01 14:20:13.092722 2016] [mpm_prefork:notice] [pid 13220] AH00163: Apache/2.4.18 (Unix) mod_wsgi/4.5.3 Python/2.7.10 configured -- resuming normal operations
[Mon Aug 01 14:20:13.092995 2016] [core:notice] [pid 13220] AH00094: Command line: 'httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8080:502/httpd.conf -E /dev/stderr -D FOREGROUND'

And that is it. Our Opal application is now running.

You may be thinking at this point that using ‘runserver’ is just as easy, so what is the point, but if you look closely at the output of ‘warpdrive start’, you will see that the Django development server is not being used. Instead Apache/mod_wsgi is being used. That is, a production grade WSGI server. Not only that, you didn’t have to configure anything, all the set up and running of Apache and mod_wsgi was done for you.

Using a different WSGI server such as uWSGI is not any harder. In the case of uWSGI, all you would need to do is create the file ‘.warpdrive/server_type’ and place in it ‘uwsgi’, to override the default of using Apache/mod_wsgi. Then run ‘warpdrive build’ and once again ‘warpdrive start’ and you will instead be running with uWSGI.

  -----> Configuring for deployment mode: of 'auto'
  -----> Default WSGI server type is 'uwsgi'
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
  -----> Running server script start-uwsgi
  -----> Executing server command 'uwsgi --master --http-socket :8080 --enable-threads --threads=5 --thunder-lock --single-interpreter --die-on-term --module mynewapp.wsgi --callable application --static-map /assets/=/Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/assets/'
[uwsgi-static] added mapping for /assets/ => /Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/assets/
*** Starting uWSGI 2.0.13.1 (64bit) on [Mon Aug 1 14:27:02 2016] ***
...

In all cases, no matter which WSGI server you are using, warpdrive will worry about ensuring the minimum sane set of options are provided to the WSGI server as well as any required for the specific WSGI application. In this case warpdrive even handled the task of making sure the WSGI server knew how to host the static files the application needs.

Initialising an application database

Our Opal application is again running and we can access it via the browser from 'http://localhost:8080/'. Do so though and we encounter a new problem though.

Exception Type: OperationalError
Exception Value: no such table: axes_accessattempt

This gets back to that magic that was being done when ‘runserver’ was being used. Specifically, the ‘runserver’ command had been set up to also automatically ensure that the database being used was initialised and that an initial account created.

Doing that for a development system is fine, but you would have to be careful about automating that in a production system. For starters, although in a development system you can use a file based database such as SQLite, in production you are more likely going to be using a database such as MySQL or PostgreSQL. These will be handling your real data and so you have to be much more careful in what you do with those databases.

When using Django, which Opal is based on, it provides two management commands for initialising a database and creating accounts. These are ‘migrate’ and ‘createsuperuser’. The ‘migrate’ command actually serves two purposes. It can be used to initialise the initial database, but also perform database migrations when the database model used by the application code changes.

These are slightly magic steps which you need to know about how Django works to know how to run. When they were automatically triggered by ‘runserver’ you didn’t have to know how to run them as that knowledge was coded into the scripts triggered by ‘runserver’.

As codifying such steps is beneficial from the stand point of ensuring that such steps are captured and always done the same way, warpdrive provides a mechanism called action hooks for recording what these steps are. You can then get warpdrive to run them for you and you don’t have to know the details. You can embed in the action as much magic as you need to, including steps like ensuring that your database is actually running before attempting anything, or allowing details of accounts to create to be supplied through environment variables or configuration files.

As an example, lets create our first action hook. This we will save away in the file ‘.warpdrive/action_hooks/setup’.

#!/bin/bash
echo " -----> Running Django database migration"
python manage.py migrate
if [ x"$DJANGO_ADMIN_USERNAME" != x"" ]; then
    echo " -----> Creating predefined Django super user"
    (cat - | python manage.py shell) << !
from django.contrib.auth.models import User;
User.objects.create_superuser('$DJANGO_ADMIN_USERNAME',
'$DJANGO_ADMIN_EMAIL',
'$DJANGO_ADMIN_PASSWORD')
!
else
if (tty > /dev/null 2>&1); then
echo " -----> Running Django super user creation"
python manage.py createsuperuser
fi
fi

This captures the steps we need to initialise the database and create an initial account. For the account creation, we can either supply the details via environment variables, or if we are running in an interactive shell, it will prompt us. Having created this action hook we can now run ‘warpdrive setup’.

  -----> Running .warpdrive/action_hooks/setup
-----> Running Django database migration
Operations to perform:
Synchronize unmigrated apps: search, staticfiles, axes, messages, compressor, rest_framework
Apply all migrations: sessions, admin, opal, sites, auth, reversion, contenttypes, mynewapp
Synchronizing apps without migrations:
Creating tables...
Creating table axes_accessattempt
Creating table axes_accesslog
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
...
Applying sites.0001_initial... OK
-----> Running Django super user creation
Username (leave blank to use 'graham'):
Email address: graham@example.com
Password:
Password (again):
Superuser created successfully.

Running ‘warpdrive start’ once again we find the application is now all working fine and we can log in with the account we created. 

The contents of the ‘setup’ script is typical here of what is required for database initialisation when using Django. The other set of actions we want to capture for Django is what needs to be done when migrating the database after database model changes. These we can capture in the file ‘.warpdrive/action_hooks/migrate’.

#!/bin/bash
echo " -----> Running Django database migration"
python manage.py migrate

Why it is better to capture these commands as action hooks and have warpdrive execute them for you, is that the commands are now a part of your application code. You don’t need to go look up some documentation to remember what the steps are. All you need to remember is the commands ‘warpdrive setup’ and ‘warpdrive migrate’.

Another important reason is that if there are any special environment variables that need to be set to replicate the actual environment when your web application is run, warpdrive will also worry about setting those as well. This means you wouldn’t need to remember to set some special value for the ‘DJANGO_SETTINGS_MODULE' environment variable in order to run the Django management commands directly. The warpdrive command will know what is required and set it up for you based on what you have captured about that in your application code.

Moving to a production environment

Using ‘warpdrive’ in our local environment has allowed us to more easily use a production grade WSGI server during development. Using the same WSGI server as we will use in production means we are more likely to pick up problems which will not show up using the development server.

The action hooks feature of warpdrive has also resulted us in capturing those important steps we need to run to initialise any database and later perform database migrations when we make changes to our database model.

That is a good start, but what now if we want to run our Opal application in a production environment?

The first example of how we might want to do that is to use Docker. For that though we first need too create a Docker image which contains our application along with any WSGI server and configuration needed to startup the application.

This step is where people often waste quite a lot of time. Developers can’t resist new toys to play with and so they feel it is imperative that they learn everything about this new Docker tool, throwing away any wisdom they may have accumulated about best practices over time and start from scratch, building up their own special Docker image piece by piece.

More often than not this results in a poorly constructed Docker image that doesn’t follow best practices and which can well be insecure, running as root and requiring it be run in a way that could easily lead to being able to break into your wider systems if someone can compromise your web application.

With warpdrive there is a much better way of moving to Docker. That is to have warpdrive build the Docker image for you. You don’t need to know anything about creating Docker images as warpdrive will build up the image ensuring that best practices are being used.

To package our Opal application up into a Docker image, all we need to do is run the ‘warpdrive image’ command.

(warpdrive+opal) $ warpdrive image opal
I0801 15:35:29.321041 14900 install.go:251] Using "assemble" installed from "image:///opt/app-root/s2i/bin/assemble"
I0801 15:35:29.321223 14900 install.go:251] Using "run" installed from "image:///opt/app-root/s2i/bin/run"
I0801 15:35:29.321280 14900 install.go:251] Using "save-artifacts" installed from "image:///opt/app-root/s2i/bin/save-artifacts"
  ---> Installing application source
  ---> Building application from source
  -----> Installing dependencies with pip (requirements.txt)
Collecting cryptography==1.3.2 (from -r requirements.txt (line 2))
Downloading cryptography-1.3.2.tar.gz (383kB)
Collecting Django==1.8.3 (from -r requirements.txt (line 3))
Downloading Django-1.8.3-py2.py3-none-any.whl (6.2MB)
...
-----> Collecting static files for Django
78 static files copied to '/opt/app-root/src/mynewapp/assets', 465 unmodified.
---> Fix permissions on application source

This should look familiar to you as in building the Docker image it is using the same ‘warpdrive build’ command that you used in your local environment. This is being done within a Docker base image which has already been set up with Python and warpdrive.

By using the same tooling, in the form of warpdrive, in your local environment as well as in constructing the Docker image, you have a better guarantee that things are being set up in the same way and will also run in the same way. This removes the disparity that usually exists between working in a local environment and what you have in your production environment.

The final result of that ‘warpdrive image’ command is that you now have a Docker image named ‘opal’ which we can be run using ‘docker run’.

(warpdrive+opal) $ docker run --rm -p 8080:8080 opal
---> Executing the start up script
-----> Configuring for deployment mode: of 'auto'
-----> Default WSGI server type is 'mod_wsgi'
Python 2.7.12 (default, Jul 29 2016, 00:52:26)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
  -----> Running server script start-mod_wsgi
  -----> Executing server command 'mod_wsgi-express start-server --log-to-terminal --startup-log --port 8080 --application-type module --entry-point mynewapp.wsgi --callable-object application --url-alias /assets/ /opt/app-root/src/mynewapp/assets/'
Server URL : http://localhost:8080/
Server Root : /tmp/mod_wsgi-localhost:8080:1001
Server Conf : /tmp/mod_wsgi-localhost:8080:1001/httpd.conf
Error Log File : /dev/stderr (warn)
Startup Log File : /dev/stderr
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : en_US.UTF-8
[Mon Aug 01 05:49:45.774485 2016] [mpm_event:notice] [pid 20:tid 140425572333312] AH00489: Apache/2.4.23 (Unix) mod_wsgi/4.5.3 Python/2.7.12 configured -- resuming normal operations
[Mon Aug 01 05:49:45.774622 2016] [core:notice] [pid 20:tid 140425572333312] AH00094: Command line: 'httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8080:1001/httpd.conf -E /dev/stderr -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

Like with how ‘warpdrive build’ was used in constructing the Docker image, the ‘warpdrive start’ command is also used in the final container when run.

We are still only using the file system database SQLite, which will not survive the life of the container, at this point, and we also need to initialise that database, but we can again use the ‘warpdrive setup’ command.

$ docker exec -it berserk_galileo warpdrive setup
-----> Running .warpdrive/action_hooks/setup
-----> Running Django database migration
Operations to perform:
Synchronize unmigrated apps: compressor, staticfiles, search, messages, rest_framework, axes
Apply all migrations: sessions, contenttypes, admin, mynewapp, sites, reversion, auth, opal
Synchronizing apps without migrations:
Creating tables...
Creating table axes_accessattempt
Creating table axes_accesslog
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
...
-----> Running Django super user creation
Username (leave blank to use 'default'): graham
Email address: graham@example.com
Password:
Password (again):
Superuser created successfully.

This is where the benefit of having captured all those steps to initialise the database in an action hook comes into play. You only need to know the one command and not all the individual commands.

Making an application production ready

As you can see using warpdrive can certainly help to simplify getting a Python web application running and getting it into production using a container runtime such as Docker. Part of the benefits of using warpdrive are that it handles the WSGI server for you, but also features like action hooks, which help to ensure you capture important key steps around how to setup your application.

There is still more to getting an application production ready than just this though. Especially when using containers, because the running container is ephemeral and so any local data is lost when the container stops, it is important to use an external database or persistent storage. Work also needs to be done around how you configure your application as well as log information from it.

In followup posts I will start to delve into these issues and how warpdrive can help you configure your application for the target environment. I will also go into detail about how warpdrive can be used with PaaS offerings such as OpenShift.

 

Tuesday, July 26, 2016

Installing mod_wsgi on MacOS X with native operating system tools.

Operating systems inevitably change over time, and because writing documentation is often an after thought or developers have no time, the existing instructions on how to install a piece of software can suffer bit rot and stop working. This has been the case for a while with various parts of the documentation for mod_wsgi. This post is a first step at least in getting the documentation for installing mod_wsgi on MacOS X brought up to date. The post will focus on installing mod_wsgi using the native tools that the MacOS X operating system provides.

Installing direct from source code

A precompiled binary package for mod_wsgi is actually available from Apple as part of the Mac OS X server app available from the MacOS X App Store. The last time I looked this was a very old version of mod_wsgi from many years ago. Unless for some reason you really need to use the version of mod_wsgi provided with MacOS X server, I would instead recommend you install an up to date version of mod_wsgi direct from source code.

Installation of mod_wsgi from source code on MacOS X used to be a simple matter, but with the introduction of System Integrity Protection in MacOS X El Capitan this has become a bit more complicated.  Lets step through the normal steps for installing mod_wsgi to see what the issue is.

After having downloaded and extracted the latest source code for mod_wsgi, to install mod_wsgi direct into an Apache installation involves running the traditional steps for most Open Source packages of doing a ‘configure’, ‘make’ and ‘sudo make install’.

Before you do that it is important though that you have at least installed the Xcode command line tools. This is an Apple supplied package for MacOS X which contains the C compiler we will need to build the mod_wsgi source code, as well as the headers files and other support files for the Apache httpd web server.

To check that you have the Xcode command line tools you can run ‘xcode-select --install’. If you have them installed already, you should see the message below, otherwise you should be stepped through installation of the package.

$ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" to install updates

Do ensure that you have run software update to get the latest version for your operating system revision if you don’t regularly update.

With the Xcode command line tools installed, you can now run the ‘configure’ script found in the mod_wsgi source directory.

$ ./configure
checking for apxs2... no
checking for apxs... /usr/sbin/apxs
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for prctl... no
checking Apache version... 2.4.18
checking for python... /usr/bin/python
configure: creating ./config.status
config.status: creating Makefile

Important to note here is that we want the ‘apxs’ version found to be ‘/usr/sbin/apxs’ and the ‘python’ version found to be ‘/usr/bin/python’. If these aren’t the versions found then it indicates that you have a Python or Apache httpd server installation which was installed separately and is not the native versions supplied with MacOS X. I am not going to cover using separate Python or Apache httpd server installations in this post and assume you only have the native tools.

The next step is to run ‘make’.

$ make
./apxs -c -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -DENABLE_DTRACE -DMACOSX -DNDEBUG -DNDEBUG -DENABLE_DTRACE -Wc,-g -Wc,-O2 -Wc,'-arch x86_64' src/server/mod_wsgi.c src/server/wsgi_*.c -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -arch x86_64 -lpython2.7 -ldl -framework CoreFoundation
./libtool --silent --mode=compile /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -DDARWIN_10 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.Internal.sdk/usr/include/apr-1 -I/usr/include/apache2 -I/usr/include/apr-1 -I/usr/include/apr-1 -g -O2 -arch x86_64 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -DENABLE_DTRACE -DMACOSX -DNDEBUG -DNDEBUG -DENABLE_DTRACE -c -o src/server/mod_wsgi.lo src/server/mod_wsgi.c && touch src/server/mod_wsgi.slo
...
./libtool --silent --mode=link /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc    -o src/server/mod_wsgi.la  -rpath /usr/libexec/apache2 -module -avoid-version    src/server/wsgi_validate.lo src/server/wsgi_thread.lo src/server/wsgi_stream.lo src/server/wsgi_server.lo src/server/wsgi_restrict.lo src/server/wsgi_metrics.lo src/server/wsgi_memory.lo src/server/wsgi_logger.lo src/server/wsgi_interp.lo src/server/wsgi_daemon.lo src/server/wsgi_convert.lo src/server/wsgi_buckets.lo src/server/wsgi_apache.lo src/server/mod_wsgi.lo -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -arch x86_64 -lpython2.7 -ldl -framework CoreFoundation

If you look closely you might note something strange in the output from running ‘make’. That is that rather than running ‘/usr/bin/apxs’ it is running a version of ‘apxs’ out of the directory where ‘make’ was run. Similarly, the system version of ‘libtool’ is ignored and a local copy used instead.

For those not familiar with what ‘apxs’ is, it is a tool supplied with the Apache httpd server package to assist in the compilation and installation of Apache modules. Unfortunately, every time that a new version of MacOS X comes out Apple somehow breaks the ‘apxs’ tool so that it doesn’t work. Typically this is because the ‘apxs’ tool embeds paths to a special variant of the C compiler used when Apple build their own packages. This is different to the C compiler which we as users can use when we install the Xcode command line tools. More specifically, the C compiler from the Xcode command line tools is installed in a different location to what ‘apxs’ expects and so it fails. A similar problem exists with ‘libtool’.

This issue with ‘apxs’ and ‘libtool’ being broken has been present for a number of MacOS X versions now and Apple seems to have no interest in fixing it. To get around the problem the ‘configure’ script of mod_wsgi creates copies of the original ‘apxs’ and ‘libtool’ programs and fixes them up so correct paths are used. This is the reason why local versions of those tools are used.

With the build of mod_wsgi now complete we just need to install it by running ‘sudo make install’. The result of this should be that the compiled ‘mod_wsgi.so’ module will be installed into the Apache httpd server installation modules directory. Because though of the System Integrity Protection feature mentioned above, this isn’t what now occurs. Instead the installation fails.

$ sudo make install
Password:
./apxs -i -S LIBEXECDIR=/usr/libexec/apache2 -n 'mod_wsgi' src/server/mod_wsgi.la
/usr/share/httpd/build/instdso.sh SH_LIBTOOL='./libtool' src/server/mod_wsgi.la /usr/libexec/apache2
./libtool --mode=install install src/server/mod_wsgi.la /usr/libexec/apache2/
libtool: install: install src/server/.libs/mod_wsgi.so /usr/libexec/apache2/mod_wsgi.so
install: /usr/libexec/apache2/mod_wsgi.so: Operation not permitted
apxs:Error: Command failed with rc=4653056
.
make: *** [install] Error 1

The rather obscure error message we get when this fails is ‘Operation not permitted’. This doesn’t exactly tell us a lot and is mighty confusing to anyone installing mod_wsgi, or any other Apache module.

The reason we get this error is that the System Integrity Protection feature means that even when running as root, it is no longer possible to copy new files into certain system directories on MacOS X. This is meant in part to protect the operating system directories from being messed up by a user, but means we are now prohibited from installing additional Apache httpd server modules into the standard modules directory of ‘/usr/libexec/apache2’.

Creating a separate modules directory

There are a few solutions to the problem that the System Integrity Protection feature causes.

Since it is the cause of the problem, you might think about disabling the System Integrity Protection feature. Although that sounds great, you really really really do not want to do this. This is part of the feature set that MacOS X uses to protect your system from malware, so disabling it is a bad idea. Do not go there nor even contemplate doing so.

The quickest solution therefore is to install the compiled ‘mod_wsgi.so’ module in a different location that we can write to and setup the Apache httpd server to reference it from that location. To do that we need only override the location using the ‘make’ variable ‘LIBEXECDIR’ when we run ‘sudo make install’. For this example we will use the directory ‘/usr/local/httpd/modules’ instead of the default on MacOS X of ‘/usr/libexec/apache2’.

$ sudo make install LIBEXECDIR=/usr/local/httpd/modules
Password:
mkdir -p /usr/local/httpd/modules
./apxs -i -S LIBEXECDIR=/usr/local/httpd/modules -n 'mod_wsgi' src/server/mod_wsgi.la
/usr/share/httpd/build/instdso.sh SH_LIBTOOL='./libtool' src/server/mod_wsgi.la /usr/local/httpd/modules
./libtool --mode=install install src/server/mod_wsgi.la /usr/local/httpd/modules/
libtool: install: install src/server/.libs/mod_wsgi.so /usr/local/httpd/modules/mod_wsgi.so
libtool: install: install src/server/.libs/mod_wsgi.lai /usr/local/httpd/modules/mod_wsgi.la
libtool: install: install src/server/.libs/mod_wsgi.a /usr/local/httpd/modules/mod_wsgi.a
libtool: install: chmod 644 /usr/local/httpd/modules/mod_wsgi.a
libtool: install: ranlib /usr/local/httpd/modules/mod_wsgi.a
libtool: install: warning: remember to run `libtool --finish /usr/libexec/apache2'
chmod 755 /usr/local/httpd/modules/mod_wsgi.so

Although the output from running this command shows a warning about running ‘libtool --finish’ you can ignore it. To be honest I am not actually sure how it even still knows about the directory ‘/usr/libexec/apache2’, but for MacOS X everything still works without doing that step.

With the mod_wsgi module installed, in the Apache httpd server configuration file you would then use:

LoadModule wsgi_module /usr/local/httpd/modules/mod_wsgi.so

rather than the normal:

LoadModule wsgi_module libexec/apache2/mod_wsgi.so

This gets us beyond the System Integrity Protection problem caused by using MacOS X El Capitan. You would then configure and set up the Apache httpd server and mod_wsgi for your specific WSGI application in the same way as you normally would.

Using mod_wsgi during development

Do note that if you only want to run mod_wsgi during development, and especially if only on a non privileged port instead of the standard port 80, you are better off installing and using mod_wsgi-express.

The benefit of using mod_wsgi-express is that it is easier to install, and gives you a command line program for starting it up, with the Apache httpd server and mod_wsgi automatically configured for you.

To install mod_wsgi-express on MacOS X you still need to ensure you have installed the Xcode command line tools as explained above, but once you have done that it is a simple matter of running:

pip install mod_wsgi

Rather than the mod_wsgi module being installed into your Apache httpd server installation, the module and ‘mod_wsgi-express’ program will be installed into your Python installation. Hosting your WSGI application with mod_wsgi can then be as simple as running:

mod_wsgi-express start-server hello.wsgi

That mod_wsgi-express installs into your Python installation makes it very easy to use mod_wsgi with different Python installations, be they different Python versions or Python virtual environments, at the same time. You can therefore much more readily run mod_wsgi for both Python 2 and Python 3 on the same system. Each mod_wsgi-express instance is distinct and would need to run on different ports, but you can if need be use your main Apache httpd server installation as a proxy in front of these if needing to make both available on the standard port 80 at the same time.

For more information on mod_wsgi-express check out the documentation on PyPi for the mod_wsgi package or read the blog post where it was introduced. I have also posted here about proxying to instances of mod_wsgi-express as well.

 

Friday, April 8, 2016

How are you using Docker in your development workflow?

If you have been reading my blog posts rather than simply flicking them out of your news feed as a bit of noise, you will know that I have been working on a project which aims to make the deployment of Python web applications easier. I wrote a bit about this in the post titled 'Building a better user experience for deploying Python web applications’.

That post got a surprising number of reads, many more than I would normally expect to see in such a short period of time. There definitely therefore seems to be a lot of interest in the topic.

For what I am developing I am targeting a local user development workflow where you work directly on your own computer, as well as then deploying direct to some host. At the same time though, I am providing a way to ease the transition to bundling up your application as a Docker image so that it can then be run using a Docker hosting service or a more comprehensive container application platform or PaaS.

In the workflow I am creating I am allowing for the ability to also iterate on changes to your code base while it is running inside of a Docker container. This is so that where your local development system is a different operating system to where it is deployed, you can still easily debug your code for the target system.

Personally I feel that most people will still likely develop on their own local system first, rather than doing development exclusively within Docker in some way.

Although this is my view, I am very much interested in how others see Docker fitting into the development workflow when implementing Python web applications. So I would like to hear your feedback so I can factor in what people are actually doing, or what they want to be able to do, within the system I am creating.

Luckily my colleague, the awesome Steve Pousty, has recently hosted up a survey asking similar questions about use of Docker in development.

It would help me immensely in what I am working on for Python if you could respond to Steve’s survey as then I can see what I can learn from the results as well. The survey is only short and should take as little as five minutes to fill in. You can take longer of course if you want to provide additional feedback on top of the short list of multiple choice questions.

When done Steve will be collating and making available the results, so it should be an interesting set of results for anyone working in this space.

You can see Steve’s original blog post about the survey at:

* Input Request: How Do You Use Docker Containers For Your Local Development?

The survey itself you can find over on Survey Monkey at:

https://www.surveymonkey.com/r/dockerdev

If you fill in the survey, make sure you mark Python in the languages you are using so I know what responses may be extra relevant to me. I would also be interested to know what Python WSGI server you are using, or whether you are using some ASYNC Python web server. So add that as extra information at the end of the survey.

In the system I am developing, I am trying to cater for all the main WSGI servers in use (gunicorn, mod_wsgi-express, uWSGI, Waitress), as well as providing ways of also running up other servers based on the ASYNC model. Knowing what servers you are using will therefore help me understand what else I should be supporting in the workflow.

Looking forward to any comments you have. Thanks.

Thursday, April 7, 2016

Learning more about using OpenShift 3.

I still have a long list of topics I could post about here on my own blog site, but over the last couple of months or so, I have been having too much fun playing with the new version of OpenShift based on Docker and Kubernetes, and understanding everything about it. The more I dig into OpenShift, the more awesome it gets as far as being the best platform around for deploying applications in our new containerised world.

A platform alone isn’t though going to give you everything you may need to provide you with the best experience for working with a particular programming language, such as Python, but this is where I am working on my own magic secret source to make everything even easier for us Python developers. I described a bit about what I was doing around improving the deployment experience for Python web applications in a prior blog post. I am going to start ramping up soon on writing all the documentation for the packages I have been working on and hope to have more to show by the time of PyCon US.

In the interim, if you are interested in OpenShift and some of the things I have been looking at and uncovering, I have been posting over on the OpenShift blog site. The blog posts which I have posted up there over the past month and a bit are:

  • Using persistent volumes with docker as a Developer on OpenShift - This explains about how to use persistent volumes with OpenShift. This is an area where OpenShift goes beyond what is possible with hosting environments which only support 12 factor or cloud native applications. That is, not only can you host up web applications, you can run applications such as databases, which require access to persistent file system based storage
  • Using an Image Source to reduce build times - This post is actually an extension to a post I did here on my own site about improving Docker build times for Python applications. In this post I show how one can include the sped up build mechanism using a Python wheelhouse within the OpenShift build and deployment infrastructure.
  • Using a generic webhook to trigger builds - This describes how to use generic web hooks to trigger a build and deployment within OpenShift. This can easily be done when using GitHub to host your web application code, but in this case I wanted to trigger the build and deployment upon the successful completion of a test when using Travis CI, rather than straight away when code was pushed up to GitHub. This necessitated implementing a web hook proxy and I show how that was done.
  • Working with OpenShift configurations - Finally, this post provides a cheat sheet for where to quickly find information about what different configuration objects in OpenShift are all about and what settings they provide.

I will be posting about more OpenShift topics on the OpenShift blog in the future, so if you are at all interested in where the next generation of Platform as a Service (PaaS), or container application platforms, are headed, ensure you follow that blog.

If you are attending PyCon US this year in Portland Oregon, you also have the opportunity to learn more about OpenShift 3. This year I will be presenting a workshop titled 'Docker, Kubernetes, and OpenShift: Python Containers for the Real World’. This is a free workshop. All you need to do if you are already attending PyCon US, is to go back to your registration details on the PyCon US web site and go into the page for adding tutorials or workshops. You will find this workshop listed there and you can add it. Repeating again, attending the workshop will not cost you anything extra, so if you are in Portland early for PyCon US then come along. I will be talking about what OpenShift is, and how it uses Docker and Kubernetes. I will also be demonstrating the deployment of a Django based web application along with a database. This will most likely be using the Wagtail CMS if you are a fan of that. Hope to see you there.

Wednesday, March 2, 2016

Speeding up Docker build times for Python applications.

I recently wrote a post where I talked about building a better user experience for deploying Python web applications. If one counts page hits as an indicator of interest in a subject then it certainly seems like an area people would like to see improvements.

In that post I talked about a system I was working on which simplified starting up a Python web server for your web application in your local environment, but also then how you can easily move to deploying that Python web application to Docker or OpenShift 3.

In moving to Docker, or OpenShift 3 (which internally also uses Docker), the beauty of the system I described was that you didn’t have to know how to create a Docker image yourself. Instead I used a package called S2I (Source to Image) to construct the Docker image for you.

What S2I does is use a Docker base image which incorporates all the system packages and the language run time environment you need for working in a specific programming language such as Python. That same Docker image also includes a special script which is run to incorporate your web application code into a new Docker image which builds off the base image. A further script within the image starts up an appropriate web server to run your web application. In the typical case, you don’t need to know anything at all about how to configure the web server as everything is done for you.

Docker build times

A problem that can arise any time you use Docker, unless you are careful, is how long it takes to actually perform the build of the Docker image for your web application. If you are making constant changes but need to rebuild the Docker image each time to test it, or redeploy it into a live environment, you could end up waiting quite a long time over the period of your work day. Decreasing the time it takes to build the Docker image can therefore be important.

The general approach usually followed is to very carefully craft your ‘Dockerfile’ so that it uses multiple layers, where the incorporation of parts which change most frequently are done last. By doing this, the fact that Docker will cache layers and start rebuilding only at the first layer changed, means you can avoid rebuilding everything every time.

This approach does break down though in various ways, especially with Python. The use of S2I can also complicate matters because it aims to construct the final image incorporating your application code, as well as all the dependent packages required by your application in a single Docker layer.

One issue with Python is the use of a ‘requirements.txt’ file and ‘pip’ to install packages. If you need to install a lot of packages and you add a single new package to the list, then all of them have to be reinstalled. Further, if those packages are being installed in the same layer as when your application code is being incorporated, as is the case with S2I, then a change to the application code causes all the packages to also be reinstalled.

So although S2I provides a really simple and clean way of constructing Docker images without you yourself needing to know how to create them, long build times are obviously not ideal.

As an example of how long a build time can be, consider the creation of a Docker image for hosting a Wagtail CMS site using Django. The ‘requirements.txt’ file in this case contains only:

Django>=1.9,<1.10
wagtail==1.3.1
psycopg2==2.6.1

Although this isn’t all that is installed. The complete list of packages which gets installed are:

beautifulsoup4==4.4.1
Django==1.9.2
django-appconf==1.0.1
django-compressor==2.0
django-modelcluster==1.1
django-taggit==0.18.0
django-treebeard==3.0
djangorestframework==3.3.2
html5lib==0.9999999
Pillow==3.1.1
psycopg2==2.6.1
pytz==2015.7
rcssmin==1.0.6
rjsmin==1.0.12
six==1.10.0
Unidecode==0.4.19
wagtail==1.3.1
wheel==0.29.0
Willow==0.2.2

Using my ‘warpdrive’ script from the previous blog post I referenced, it can take over 5 minutes over my slow Internet connection to bring down all the required Python packages, build them and construct the image.

(warpdrive+wagtail-demo-site) $ time warpdrive image wagtail
I0301 22:01:01.374459 16060 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
I0301 22:01:01.374643 16060 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
I0301 22:01:01.374674 16060 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
---> Installing application source
---> Building application from source
-----> Installing dependencies with pip (requirements.txt)
Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
Downloading Django-1.9.2-py2.py3-none-any.whl (6.6MB)
Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
Downloading wagtail-1.3.1-py2.py3-none-any.whl (9.0MB)
Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
Downloading psycopg2-2.6.1.tar.gz (371kB)
...
Installing collected packages: Django, djangorestframework, Unidecode, Pillow, rcssmin, rjsmin, six, django-appconf, django-compressor, Willow, html5lib, django-taggit, pytz, django-modelcluster, beautifulsoup4, django-treebeard, wagtail, psycopg2
...
Running setup.py install for psycopg2: finished with status 'done'
Successfully installed Django-1.9.2 Pillow-3.1.1 Unidecode-0.4.19 Willow-0.2.2 beautifulsoup4-4.4.1 django-appconf-1.0.1 django-compressor-2.0 django-modelcluster-1.1 django-taggit-0.18.0 django-treebeard-3.0 djangorestframework-3.3.2 html5lib-0.9999999 psycopg2-2.6.1 pytz-2015.7 rcssmin-1.0.6 rjsmin-1.0.12 six-1.10.0 wagtail-1.3.1
-----> Collecting static files for Django
...
Copying '/opt/warpdrive/demo/static/js/demo.js'
...
Copying '/usr/local/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/img/gis/move_vertex_off.svg'
179 static files copied to '/home/warpdrive/django_static_root'.
---> Fix permissions on application source
real 5m40.780s
user 0m0.850s
sys 0m0.115s

If you were running ‘pip’ in your local environment and installing into a Python virtual environment, rerunning ‘pip’ on the ‘requirements.txt’ wouldn't be a big issue. This is because the packages would already be detected as being installed and so wouldn’t need to be downloaded and installed again. Even if you did blow away your Python virtual environment and recreate it, the downloaded packages would be in the cache that ‘pip’ maintains in your home directory. It could therefore just use those.

When creating Docker images however, you don’t get the benefit of those caching mechanisms because everything is done over every time. This means that all the packages have to be downloaded every time.

Using a wheelhouse

A possible solution to this is to create a wheelhouse. That is, you use ‘pip’ to create a directory of the packages you need to install as Python wheels. For pure Python packages these would just be that code, but if a Python package included C extensions, the Python wheel file would include the compiled code as object files. This means that the code doesn’t need to be recompiled every time and can simply be copied into place.

Although this can be done, working this into how you build your Docker images can get a bit messy as shown by Glyph in a blog post he wrote about it. It is therefore an area which is ripe for being simplified and so I have also been working that into what I have been doing with trying to simplify the deployment of web applications. In this post I want to show how that is progressing.

First step now is to create a special Docker image which acts as our Python wheelhouse. This can be done by running the following command.

(warpdrive+wagtail-demo-site) $ warpdrive image --build-target wheelhouse wagtail-wheelhouse
I0301 23:03:32.687290 17126 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
I0301 23:03:32.687446 17126 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
I0301 23:03:32.687475 17126 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
I0301 23:03:32.709215 17126 docker.go:286] Image "wagtail-wheelhouse:latest" not available locally, pulling ...
---> Installing application source
---> Building Python wheels for packages
-----> Installing dependencies as wheels with pip (requirements.txt)
Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
Downloading Django-1.9.2-py2.py3-none-any.whl (6.6MB)
Saved ./.warpdrive/wheelhouse/Django-1.9.2-py2.py3-none-any.whl
Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
Downloading wagtail-1.3.1-py2.py3-none-any.whl (9.0MB)
Saved ./.warpdrive/wheelhouse/wagtail-1.3.1-py2.py3-none-any.whl
Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
Downloading psycopg2-2.6.1.tar.gz (371kB)
...
---> Fix permissions on application source

This command is going to run a bit differently to the command above. Rather than use ‘pip install’ to install the actual packages, it will run ‘pip wheel’ to create the Python wheels we are after. At that point it will stop, as we don’t need it do additional steps such as run ‘collectstatic’ for Django to gather up static file assets. This will still take up to 5 minutes though since the bulk of the time was involved in downloading and building the packages.

Once we have our wheelhouse, when building the Docker image for our application, we can point it at the wheelhouse as a source for the prebuilt Python packages we want to install. We can even tell it to take what it provides as the authority and not consult the Python package index (PyPi) to check whether there aren’t newer versions of the packages when packages haven’t been pinned to a specific package.

(warpdrive+wagtail-demo-site) $ time warpdrive image --wheelhouse wagtail-wheelhouse --no-index wagtail
warpdrive-image-17312
I0301 23:12:54.610882 17329 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
I0301 23:12:54.611033 17329 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
I0301 23:12:54.611089 17329 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
---> Installing application source
---> Building application from source
-----> Found Python wheelhouse of packages
-----> Installing dependencies with pip (requirements.txt)
Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
...Installing collected packages: Django, Unidecode, pytz, django-modelcluster, djangorestframework, Pillow, django-treebeard, django-taggit, six, Willow, rjsmin, django-appconf, rcssmin, django-compressor, beautifulsoup4, html5lib, wagtail, psycopg2
Successfully installed Django-1.9.2 Pillow-3.1.1 Unidecode-0.4.19 Willow-0.2.2 beautifulsoup4-4.4.1 django-appconf-1.0.1 django-compressor-2.0 django-modelcluster-1.1 django-taggit-0.18.0 django-treebeard-3.0 djangorestframework-3.3.2 html5lib-0.9999999 psycopg2-2.6.1 pytz-2015.7 rcssmin-1.0.6 rjsmin-1.0.12 six-1.10.0 wagtail-1.3.1
-----> Collecting static files for Django
...
Copying '/opt/warpdrive/demo/static/js/demo.js'
...
Copying '/usr/local/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/img/gis/move_vertex_off.svg'
179 static files copied to '/home/warpdrive/django_static_root'.
---> Fix permissions on application source
real 0m45.859s
user 0m3.555s
sys 0m2.575s

With our wheelhouse, building of the Docker image for our web application has dropped from over 5 minutes down to less than a minute. This is because when installing the Python packages, it is able to reuse the pre built packages from the wheelhouse. This means a quicker turnaround for creating a new application image. We will only need to rebuild the wheelhouse itself if we change what packages we need to have installed.

Incremental builds

Reuse therefore allows us to speed up the building of Docker images considerably where we have a lot of Python packages that need to be installed. The reuse of previous builds can also be used in another way, which is to reuse the prior wheelhouse itself when updating the wheelhouse after changes to the list of packages we need.

(warpdrive+wagtail-demo-site) $ time warpdrive image --build-target wheelhouse wagtail-wheelhouse
I0301 23:18:24.150533 17448 install.go:236] Using "assemble" installed from "image:///usr/local/s2i/bin/assemble"
I0301 23:18:24.151074 17448 install.go:236] Using "run" installed from "image:///usr/local/s2i/bin/run"
I0301 23:18:24.151121 17448 install.go:236] Using "save-artifacts" installed from "image:///usr/local/s2i/bin/save-artifacts"
---> Restoring wheelhouse from prior build
---> Installing application source
---> Building Python wheels for packages
-----> Installing dependencies as wheels with pip (requirements.txt)
Collecting Django<1.10,>=1.9 (from -r requirements.txt (line 1))
File was already downloaded /opt/warpdrive/.warpdrive/wheelhouse/Django-1.9.2-py2.py3-none-any.whl
Collecting wagtail==1.3.1 (from -r requirements.txt (line 2))
File was already downloaded /opt/warpdrive/.warpdrive/wheelhouse/wagtail-1.3.1-py2.py3-none-any.whl
Collecting psycopg2==2.6.1 (from -r requirements.txt (line 3))
Using cached psycopg2-2.6.1.tar.gz
...
---> Fix permissions on application source
real 1m17.180s
user 0m3.653s
sys 0m2.316s

Here we have run the exact same command as we ran before to create the wheelhouse in the first place, but instead of taking 5 minutes to build, it has taken just over 1 minute.

This speed up was achieved because we were able to copy across the ‘pip’ cache as well as the directory of Python wheel files from the previous instance of the wheelhouse.

Not a Dockerfile in sight

Now what you didn’t see here at all was a ‘Dockerfile’. For me this is a good thing.

The problem with Docker right now is that the novelty still hasn’t warn off, with it still not being seen for what it is, just another tool we can use. As a result we are still in this phase where developers using Docker like to play with it and so try and do everything themselves from scratch. We need to get beyond that phase and start incorporating best practices into canned scripts and systems and simply get on with using it.

Anyway, this is where I am at least heading with the work I am doing. That is, encapsulate all the best practices for Python web application deployment, including the building of Docker images which you can run directly, or with a PaaS using Docker such as OpenShift. The aim here being to make it so much easier for you, with you knowing that you can trust that the mechanisms have been put together will all the best practices being followed. After all, do you really want to keep reinventing the wheel all the time?