Monday, October 22, 2018

Packaging mod_wsgi into a zipapp using shiv.

At the recent DjangoCon US conference, Peter Baumgartner presented a talk titled Containerless Django: Deploying without Docker. In the talk Peter described what a zipapp (executable Python zip archive) is and how these could be created using the shiv tool, the aim being to be able to create a single file executable for a complete Python application, including all its Python package dependencies. The only requirement would be that any target platform you copy the executable to, must provide a matching Python runtime.

A question which was asked after the talk was whether it would be possible to use shiv in conjunction with mod_wsgi. My initial thinking was that yes, it should be possible. As it turned out, it wouldn't work as things were, due to how shiv set up the application so it knew where all the Python package dependencies were unpacked when the executable was run. With a bit of black magic though, I was able to get mod_wsgi to work. This post is to highlight what what can be done in case you want to try it out for yourself.

Goals of zipapp and shiv

The goal of the zipapp executable format is to allow a Python application, along with all its Python package dependencies, to be bundled up as a single executable file. The zipapp format and support for it has been available since Python 2.6, but probably isn't that widely known about. The format got additional support in Python 3.5 via PEP 441. It still though had limitations, including not being able to be used for Python packages with C extensions.

The shiv tool for creating zipapp executables, works around this and other issues by intercepting the entry point for the zipapp executable and unpacking files from the executable into the file system before actually running the bundled application.

As an example of what you can do, consider the certbot tool for managing creation of certificates using Let's Encrypt.

To install certbot is easy enough and can be done with pip. The problem is, that in addition to its own code, it also depends on a number of other Python packages. Best practice would be to install certbot and those packages into their own Python virtual environment. You could do this yourself, or you could use pipsi. Neither of these makes it easy to then copy the application to another host. This is where shiv can come into play.

For this case of certbot, to create a single executable file for certbot, you can run shiv as:

$ shiv certbot -o certbot.pyz -c certbot

The result is a single certbot.pyz file, which you can run like any other executable.

$ ./certbot.pyz --help

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. The most common SUBCOMMANDS and flags are ...

Being a single file, which includes both the code for certbot itself, and all the Python packages it depends on, it is easy to copy the file to another host. The only requirements are that it has the same version of Python installed, and the target host is the same architecture if Python packages with C extensions are used.

Trying shiv with mod_wsgi

Trying this with mod_wsgi-express things weren't quite as successful. The building of the executable file ran okay:

$ shiv 'mod_wsgi==4.6.4' -o mod_wsgi-express.pyz -c mod_wsgi-express

But running:

$ ./mod_wsgi-express.pyz start-server --log-to-terminal

yielded the error:

File "/var/tmp/mod_wsgi-localhost:8000:501/handler.wsgi", line 7, in 
     import mod_wsgi.server
ModuleNotFoundError: No module named 'mod_wsgi.server'; 'mod_wsgi' is not a package

This fails because when mod_wsgi-express start-server is run, it is actually performing a fork/exec of the Apache httpd server, but the directory shiv has created holding all the required Python packages, isn't known of by the Apache mod_wsgi module when it is in turn loaded. It therefore cannot find the Python module called mod_wsgi which the handler.wsgi file for mod_wsgi-express uses.

The specific reason this is the case, is because the shiv bootstrap code executed when the executable file is run, only adjusts sys.path of the current process so that the bundled Python packages can be found. It does not set the PYTHONPATH environment such that it would be inherited by subprocesses. The result is that the directory is only known of by that process alone, or any direct forks. It will not be known of by any sub process created by executing a standalone program such as the Apache httpd server.

A bit of black magic

Unfortunately, it doesn't seem that the shiv bootstrap code leaves any global state variables in a readily accessible place, which can be used to determine whether shiv was used, and what the name of the directory with the required Python packages are. The only place a variable exists with the name of the directory we want, is in a local stack frame from a prior function call.

Getting access to the directory therefore relies on a bit of black magic.

    site_packages = []

    if '_bootstrap' in sys.modules:
        bootstrap = sys.modules['_bootstrap']
        if 'bootstrap' in dir(bootstrap):
            frame = inspect.currentframe()
            while frame is not None:
                code = frame.f_code
                if (code and code.co_filename == bootstrap.__file__ and
                        code.co_name == 'bootstrap' and
                        'site_packages' in frame.f_locals):
                frame = frame.f_back

This makes a guess that shiv was used by looking to see if the _bootstrap module had been imported and that it contained a bootstrap() function. If it was, we look back through the function stack looking for that function and extract the value of the site_packages variable from the locals of that function.

Knowing that shiv was used, and what the directory with the required Python packages was, we can use it in the generated configuration for Apache/mod_wsgi, so the embedded Python interpreter it runs, can find them.

Sure this is a hack, but unless shiv provides a better way of knowing when it is being run and what the directory with the Python packages was, we don't have much choice. Good thing at least is you don't have to care, as this fiddle is hidden away in mod_wsgi-express and you just need to use the right version of the mod_wsgi package, which is version 4.6.5 or newer.

$ shiv 'mod_wsgi==4.6.5' -o mod_wsgi-express.pyz -c mod_wsgi-express

Apache httpd server

When you use shiv to create the executable zipapp file, the target host must still have Python installed as well. This is because it is only the application code and Python packages it requires that are bundled in the executable file. The Python interpreter itself is not included.

In the case of mod_wsgi-express, we also need the Apache httpd server. With the above command, both the host where the executable is built and the target host must have it installed, and in the same location.

Unlike with the Python interpreter, there is a way around this for the Apache httpd server, and it can instead be bundled in the executable as well. This is by virtue of a little known companion package for mod_wsgi called mod_wsgi-httpd. Like with mod_wsgi, the mod_wsgi-httpd package exists on PyPi and can be installed using pip. When installed, it will build the Apache httpd server from source code. When mod_wsgi is then subsequently installed, it will use it instead.

$ shiv 'mod_wsgi-httpd==2.4.35.*' 'mod_wsgi==4.6.5' -o mod_wsgi-express.pyz -c mod_wsgi-express

We now have the Apache httpd server and mod_wsgi-express bundled in the one executable file. This can be copied to another host and the Apache httpd server doesn't need to be installed separately on either host.

Bundling your application

The example above for mod_wsgi-express is only bundling it and the Apache httpd server, it isn't including your application. The shiv documentation provides a few examples of bundling your application code.

If you do that, and it is a Django application, you will want to use the Django management command integration for mod_wsgi-express. You can then setup the shiv entrypoint to run the runmodwsgi Django management command.

For other WSGI applications, you can have the shiv entrypoint import mod_wsgi.server and call the start() function within that module, passing as a list the same options as you would pass to mod_wsgi-express start-server.

If want more details on this, for now would suggest posting on the mod_wsgi mailing list and I can provide additional information.

Friday, April 13, 2018

Book #2: Deploying to OpenShift

I have been more than a bit busy over the past year and this blog has become somewhat neglected. One of the reasons for being so busy was that I was working on a second book. As with the first book, which I co-authored, this book is on OpenShift. This time I am the sole author, and the book somewhat thicker, so you can imagine it has taken a fair bit of time and effort.

For those who may not know what OpenShift is, it is recognised as the go-to distribution of Kubernetes for the enterprise. Like how the Linux kernel is much more useful when it's packaged into a Linux distribution such as RHEL, Debian or Ubuntu, OpenShift takes Kubernetes and adds the extra components required to turn it into a complete container platform for running your containerised workloads.

Another way of putting it is that OpenShift is a platform to help you develop and deploy applications across a cluster of machines at scale. These can be public facing web applications, or backend applications, including micro services or databases. Applications can be implemented in any programming language you choose. The only requirement is that the application can run within a container. In terms of cloud service computing models, OpenShift implements the functionality of both a Platform as a Service (PaaS) and a Container as a Service (CaaS).

Using OpenShift as a CaaS, you can bring a pre-existing container image built to the Open Container Initiative (OCI) Image Specification (image-spec) and deploy it. The PaaS capabilities of OpenShift build on top of the ability to deploy a container image, by providing a way for you to build in OpenShift your own container image direct from your application source code and have it deployed. The application source code can consist of a Dockerfile with instructions to build a container image. Or, you can use a Source-to-Image (S2I) builder, which takes your application source code and converts it into a container image for you, without you needing to know how to write instructions for building a container image.

If that sounds interesting and you would like to read the book, the good news is that the book is provided free, compliments of Red Hat, where I work as a developer advocate for OpenShift.

To download the free electronic version, you can visit:

The book aims to be a practical guide which describes in detail how OpenShift, building on Kubernetes, enables you to automate the way you create, ship, and run applications in a containerized environment, be they cloud-native applications, or more traditional stateful applications.

In the book, you will learn the following concepts:

  • Create a project and deploy pre-existing application container images
  • Build application container images from source and deploy them
  • Implement and extend application image builders
  • Use incremental and chained builds to accelerate build times
  • Automate builds by using a webhook to link OpenShift to a Git repository
  • Add configuration and secrets to the container as project resources
  • Make an application visible outside the OpenShift cluster
  • Manage persistent storage inside an OpenShift container
  • Monitor application health and manage the application lifecycle

I hope you enjoy this free book and that it will help you get up and running on OpenShift. If looking for an environment to try out OpenShift, you can sign up for the free OpenShift Online Starter environment hosted by Red Hat.

BTW, this is the fourth book made available by Red Hat about OpenShift. I am quite proud to be able to say that every one of those books had an author from Australia. To carry through on the Australian flavour to the books, for this book the cover animal is the Australian Sulphur-Crested Cockatoo. These birds are a frequent visitor to my home. Unfortunately the picture doesn't show their true larrikin like character.

Tuesday, January 30, 2018

The "Decorator Pattern" versus the Python "wrapt" package.

Brandon Rhodes published a post today about the Decorator Pattern and how that translates into Python. He explains the manual way that the pattern can be implemented in Python as a wrapper, as well as how you can try to minimise the amount of work you need to do by overriding special methods of a Python object.

The wrapt package I authored was purpose built for this task of creating wrappers which Brandon describes, and much more. To avoid some of the name confusion around Decorator Pattern versus Python decorators, which Brandon highlights as an issue, I tend to refer to the wrappers as transparent object proxies.

Lets have a quick look at some of the examples Brandon gave and see how they would be implemented using the wrapt package and what happens when one tries to perform introspection on an object via the wrapper.

Implement: Dynamic Wrapper

Jumping to the example of the dynamic wrapper that Brandon gave, the equivalent using wrapt would be:

import wrapt
class WriteLoggingFile(wrapt.ObjectProxy):
    def __init__(self, wrapped, logger):
super(WriteLoggingFile, self).__init__(wrapped)
self._self_logger = logger
    def write(self, s):
self._self_logger.debug('wrote %s bytes to %s', len(s), self.__wrapped__)
    def writelines(self, strings):
if self.closed:
raise ValueError('this file is closed')
for s in strings:

All that needed to be provided was the methods you want to override. All that boilerplate functionality of the special methods for attribute access and update, and object iteration etc, are provided by the wrapt.ObjectProxy base class that the wrapper inherits from.

Now lets look at what happens when we introspect an instance of the wrapper object.

>>> import sys, logging
>>> stdout = WriteLoggingFile(sys.stdout, logging)
>>> dir(stdout)
['__class__', '__delattr__', '__doc__', '__enter__', '__exit__',
'__format__', '__getattribute__', '__hash__', '__init__', '__iter__',
'__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
'__sizeof__', '__str__', '__subclasshook__', 'close', 'closed',
'encoding', 'errors', 'fileno', 'flush', 'isatty', 'mode', 'name',
'newlines', 'next', 'read', 'readinto', 'readline', 'readlines',
'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines',

We get what we want to see, which is the same as what we would get if we introspect the wrapped object.

The wrapt.ObjectProxy class does much more than that though. Take for example the following:

>>> isinstance(stdout, type(sys.stdout))
>>> isinstance(stdout, file)

The isinstance() check will also succeed and say that the wrapper is an instance of the type of object which was wrapped.

It should be noted that you can't completely fool Python though:

>>> type(stdout)
<class '__main__.WriteLoggingFile'>

But then, if you want to allow for duck typing, you should never directly compare types and should always use isinstance().

Hack: Monkey-patch each object

The next example which has an equivalent when using wrapt is monkey patching an instance of an object rather than use a wrapper. Using wrapt this would be written as:

def bind_write_method(logger):
def write_and_log(wrapped, instance, args, kwargs):
wrapped(*args, **kwargs)
logger.debug('wrote %s bytes to %s', len(args[0]), instance)
return write_and_log
f = open('/dev/null', 'w')
f.write = bind_write_method(logging)(f.write)

The @wrapt.function_wrapper is a factory for creating a wrapper function. If you have used wrapt before, it does the same job as @wrapt.decorator, but doesn't have as many of the features for customisation that the latter does. When doing monkey patching, using @wrapt.function_wrapper is less confusing naming wise as well.

Using wrapt to do this in this way, introspection even still works correctly on the patched method.

>>> f.write.__name__
>>> inspect.getargspec(f.write)
ArgSpec(args=['self', 'text'], varargs=None, keywords=None, defaults=None)

Not just for Python decorators

As shown above, although the wrapt package is probably more well known as being useful for implementing well behaved Python decorators, the primary reason it was created was for implementing the Decorator Pattern for use in monkey-patching Python code dynamically.

Monkey-patching is often regarded as a hack with opinion being that it should never be used. It is absolutely essential though if you want to dynamically instrument Python code to do things like collect metric data without you needing to modify code yourself. In this situation where you would want to use it on production applications, you want to ensure the wrappers work as correctly as properly. That is what the wrapt package aims to do, ensuring as much as possible that all works properly, even in the many obscure corners cases.

If you still think this is a bad idea and don't trust what wrapt does, you may want to look under the covers of how the two leading application performance monitoring services for Python web applications instrument Python code. Hint, they use wrapt.

If you want to learn more about wrapt, check out the documentation:

I have also written over a dozen related blog posts on decorators and monkey patching:
Finally, I have presented at a number of conferences on wrapt (but not PyCon US).
I have neglected wrapt a little of late and there are a few outstanding issues that need to be addressed. If you are using wrapt, please let me know via Twitter as getting such messages is always a good motivating force when you work on open source projects. Without such messages it is too easy to get the opinion that no one is using your software and so why you should bother continuing with it.

Sunday, April 30, 2017

Deploying Jupyter Notebooks in a hosted environment.

The popularity of a programming language can often be dictated by the existence of a killer application. One example is PHP and the web site creation tool Wordpress. In the Python language community, it is harder to point to any one application that helps in promoting the language above and beyond any others. This is in part because Python can be applied in various ways and is not focused on just one area, such as implementing web applications, as is the case of PHP. Python therefore has a number of candidates for what could potentially be deemed a killer application or enabling framework, but in different subject domains.

One example of such an application for Python is the interactive code based environment provided by Jupyter Notebooks. In the data science, research and education fields, Jupyter Notebooks are becoming an increasing popular method for working with data to perform ad-hoc analysis, as well as for teaching of programming language concepts and algorithms.

Running in containers

As a teaching tool in education, Jupyter Notebooks provide a rich environment in which students can work, but one of the challenges is ensuring that all students have the same environment. This is made difficult due to students running different operating systems, or different versions of software, both bundled and self installed.

A solution to this problem is to run software in a defined environment. That is, create a software image which includes all the required packages already pre-installed in an operating system image and run that.

In the past one would have done this using a virtualised environment, where an image is run in its own virtual machine, complete with a running operating system of its own. These days, one would instead run the application in a light weight container, where operating system services are provided by the underlying host operating system and are not actually running inside of the container.

Hosted vs running locally

Distributing your own image means that if teaching with Jupyter Notebooks, you can ensure that all students are using the same software. You can also bundle any required course material as part of the image so you know that students have everything they need.

In distributing an environment in this way, although you know it will be the same for each student, it doesn't avoid the issue that students still need to install some software locally to run it. This is because they would still need to install any software that allows them to run the container image. There is also the problem of the size of images and a students access to a good enough Internet connection to download them.

The alternative to running the image locally is to use a hosted environment. This is where whoever is teaching the students would set up a purpose built hosted environment specifically to run the Jupyter Notebooks for the students.

Doing this isn't necessarily a trivial task, but at least it has to only be done once, notionally by those who would have the knowledge of how to set it up and run it, or at least the impetus to do it.

The task of running a hosted environment for Jupyter Notebooks is also made easier using the JupyterHub software, a purpose built application for interacting with users and starting up Jupyter Notebook instances on their behalf.

The JupyterHub software even supports different backend systems for running Jupyter Notebooks, including running them all on the one host, or delegating them to run across a cluster of hosts.

Generic hosting platform

Even using JupyterHub, you still generally need purpose built infrastructure to be set up and dedicated to the specific requirement of running the Jupyter Notebook instances. It is not really possible to make use of a generic multi user hosting service to run JupyterHub and have it manage the Jupyter Notebook instances. You cannot for example signup to Heroku or Amazon Elastic Beanstalk and hope to run JupyterHub on those in order to provide an environment to run a class with many students.

The closest you can get is to use the dockerspawner plugin for JupyterHub and run it across a set of hosts on a Docker Swarm cluster, or use kubespawner and run it on a Kubernetes cluster.

Neither of these though are multi user generic hosting services. Yes, you could setup and run Docker Swarm or Kubernetes, but the instance is still effectively dedicated to you. Neither Docker Swarm or Kubernetes alone are a multi tenant hosting service where the instance is shared by other users, running completely different applications all isolated from each other.

You might be thinking, why does it matter if it is my own cluster and no one else is using it for anything else. From the perspective of an educational institution or organisation wanting to run it, a key reason is cost.

When you have a dedicated cluster just for your specific use in running JupyterHub, it means one more piece of infrastructure that the IT team has to manage on top of everything else they have to run, adding to maintenance and support overheads. Also, your cluster is likely going to be very much under utilised, with machines sitting idle most of the time. Because it is dedicated infrastructure, the IT department can't readily utilise it for anything else to make the most productive use of any hardware resources.

This is where being able to run JupyterHub on a generic multi tenant hosting platform can be beneficial. The IT department need only set up one set of infrastructure which provides hosting for many different users, for different applications and purposes, at the same time. This reduces the number of systems they need to manage, as well as ensure they can make the most of available hardware resources. This all comes together to allow them to reduce overall operating expenditure and money can be used for other purposes.

Conferences and blog posts

With that as background, for the past year and a half I have as a personal side project, been researching the problem of best methods for deploying Jupyter Notebooks, directly and via JupyterHub, using container based deployment methods. This has been as an extension of other research I have been doing for many years on deploying Python web applications.

I have published a few blog posts in the past related to what I have been doing, but they have mainly been addressing specific issues which arise in running Python web applications, including Jupyter Notebooks, in containers. To date I haven't really posted much about the bigger problem I am trying to solve in running Jupyter Notebooks and JupyerHub on a modern container platform. I have submitted talks for half a dozen different conferences in that time on the topic, but have yet to have any success in getting a talk accepted.

Despite having no success in getting word out via conferences on what I have been working on, I don't like to work on things and then move on without saying something about it, so I am going to fall back to writing further blog posts about the topic. The only downside with blog posts is I do like to try and cover all the intricacies of problems in details. I can therefore go down a bit of a meandering path in getting to the final picture.

If you can put up with that you should be good, and hopefully as well as the core problem I am trying to solve, you will learn about some side topics in the process.

With that said, I present the first seven blog posts I have written on the topic. These were posted over the past few weeks, albeit not here on my own blog site, but on the blog site for OpenShift.

In this first series of blog posts I look at the issue of deploying single Jupyter Notebook instances to OpenShift. The posts are:

The final goal for all the posts I will write, is to show a simple way to deploy JupyterHub to OpenShift and run it at scale to host large numbers of students in a teaching environment. This could be your own OpenShift cluster which you have installed using the Open Source upstream OpenShift Origin project. It could also be a managed or public OpenShift instance such as OpenShift DedicatedOpenShift Online or any service from the various hosting providers starting to pop up which are using OpenShift. Finally it might be a supported on premise instance of OpenShift Container Platform.

I also hope to eventually post about any additional lessons learnt in trying to deploy JupyterHub to OpenShift in support of running a course for up to 150 students in a university setting, something which I have recently started working with a university to achieve.

Mosts posts will be made on the OpenShift blog site, but I will do followups here occasionally to summarise each series of posts and provide links. I expect it to take a few months to get through everything I want to cover.

Friday, December 9, 2016

Backdoors for becoming root in a Docker container.

In my last post, the main issue I looked at was whether you can trust what a Docker-formatted image says about the user it will run as. What we found was that if the ‘USER’ statement is used in a Dockefile, but is set to a name, you have no idea what UNIX user ID the application in the container will run as. This is because the name could be mapped to any user ID by the UNIX passwd file.

Setting up the UNIX passwd file such that a user name other than ‘root’ also mapped to the UID of 0 provided a backdoor to becoming root in the running container. By requiring that an integer UID be used with the ‘USER’ statement in a Dockerfile, we can inspect the image metadata and decide not to run the image if ‘USER’ wasn’t a non zero integer value.

Is this enough to protect us though? Are there other backdoors for becoming ‘root' in a Docker container. The answer to that is that there is, and this post will look at some of these ways.

Creating a setuid executable

The primary path for switching from a non privileged user to the ‘root’ user on a UNIX system is a setuid executable. This is an executable that has been blessed in such a way that instead of running as the user that ran it, it runs as the user who is the owner of the executable. Such setuid executables will also work inside of a Docker container.

To illustrate how a setuid executable works, lets look at the UNIX utility called ‘id', which is normally used to display information about what user and group the invoking process runs as. If run normally in our Docker container, we might see:

$ id
uid=1001(app) gid=1001(app) groups=1001(app)

Lets create a setuid version of the executable which is owned by ‘root' and bundle that in our image.

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN cp /usr/bin/id /usr/bin/id-setuid-root
RUN chmod 4711 /usr/bin/id-setuid-root
USER 1001

Running the original version of ‘id’ and the setuid version, we get:

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ id-setuid-root
uid=1001(app) gid=1001(app) euid=0(root) groups=1001(app)

As can be seen, the result is that although the real user ID is the same, the effective user ID is that of the ‘root’ user. This means that by using a setuid executable, we gain the rights to run something as if we are ‘root’, or at least very close to being ‘root’. I say very close to being ‘root’ as it is only the effective user ID which is ‘root’ and not the real user ID. In most cases it doesn’t matter, but it hardly matters anyway, as we could also switch our real identify to the ‘root’ user relatively easily from a custom setuid executable of our own.

Running programs as 'root'

In the above example we took an existing executable and made it setuid as the ‘root’ user. We can’t go and do this for every executable we want to run as ‘root’, so what do we do if we want to run an arbitrary executable as ‘root’?

You might think that is simple. All we need to do is make a copy of ‘/bin/bash’ and make it setuid as the ‘root’ user. If we can then run that, we can become ‘root’ and run any program we want as the ‘root’ user.

So this time to create the image we use:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN cp /bin/bash /bin/bash-setuid-root
RUN chmod 4711 /bin/bash-setuid-root
USER 1001

Running our setuid version of bash though, we don’t get what we expected.

bash-4.2$ id
uid=1001(app) gid=1001(app) groups=1001(app)
bash-4.2$ bash-setuid-root
bash-setuid-root-4.2$ id
uid=1001(app) gid=1001(app) groups=1001(app)

This doesn’t work because modern implementations of shells have checks builtin which look for the specific case of where they are executed with an effective user ID of ‘root’, but a non ‘root’ real user ID. In this case, just to make it harder to use this sort of backdoor, they will revert back to running as the real user ID for the effective user ID.

Since this doesn’t work, lets look at how we would achieve the same thing if we weren’t trying to use a backdoor.

The first method we would normally use to execute a command as the ‘root’ user when we are not a privileged user, is to use the ‘sudo’ command. An alternative is to use the ‘su’ command to login as the ‘root’ user.

Because in a Docker image we can install and configure anything we want, there is no reason why we can’t just set these up and use them.

FROM centos:centos7
RUN yum install -y sudo
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
# Allow anyone in group 'app' to use 'sudo' without a password.
RUN echo '%app ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
# Set the password for the 'root' user to be an empty string.
RUN echo 'root:' | chpasswd
USER 1001

With this we can very easily get an interactive shell as the ‘root’ user using ’sudo’, not even requiring a password.

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ sudo -s
# id
uid=0(root) gid=0(root) groups=0(root)

We can also just login as the ‘root’ user, supplying our empty password.

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ su root
# id
uid=0(root) gid=0(root) groups=0(root)

In both cases the real user ID is that of the ‘root’ user and not just the effective user ID.

So we didn’t even need to fiddle with a backdoor, we can just use the existing features of the operating system. We just need to install the ‘sudo’ package and configure it, or set the ‘root’ password. As it happens, both these mechanisms rely on a setuid executable, but combine it with configuration to guard against who can access them. It is a simple matter though to enable that access given that during the build of a Docker image you can change anything.

You can’t completely block 'root'

You might be thinking at this point that if we can become the ‘root’ user in these ways, what is the point then of using a check on what ‘USER’ specified for the image in the first place. Someone can always set it as a non ‘root’ user, using an integer UID to avoid any restriction on using the image, but then use a custom built backdoor marked as a setuid executable, or using existing system tools such as ‘sudo’, or ‘su’.

What is important to understand is that good security is based on having many layers. You don’t rely on just a single security measure to protect your system. Each extra layer you can add, acts as an obstacle to someone reaching their end goal. Not allowing images to run that don’t set ‘USER’ to a non zero integer ID, would be just one step you can take in a overall security plan.

So it isn’t a waste of time just yet. This is because, although there are ways of becoming the ‘root’ user even if ‘USER’ did not originally declare the container should run as ‘root’, we can still control what the ‘root’ user is actually able to do. This is achieved using Linux capabilities, and is the next layer of defence you should employ.

In the next blog post I will look at Linux capabilities and how to use Docker to restrict what someone could do even if they become the ‘root’ user.

Thursday, December 1, 2016

What USER should you use to run Docker images.

If you follow this blog and my rants on Twitter you will know that I often complain about the prevalence of Docker-formatted container images that will only work if run as the root user, even though there is no technical reason to run them as root. With more and more organisations moving towards containers and using these images in production, some at least are realising that running them as root is probably not a good idea after all. As such, organisations are for their own images at least, starting to create basic guidelines for their developers to follow around what user an image should run as.

A typical example of the most basic guidelines you can find are:

  1. Create a new UNIX group called ‘app’ with a group ID (gid) of 1001.
  2. Create a new UNIX account with user name ‘app’ with a user ID (uid) of 1001, with it being a member of the group ‘app’ and where the home directory of this user is the directory ‘/app’.
  3. Put all your application source code under the ‘/app’ directory.
  4. Set the working directory for any application run to the ‘/app’ directory.
  5. Set the user that the image will run as to the ‘app’ user.

All looks good, and better than running as the root user you might be thinking. Unfortunately there are still a number of problems with these guidelines, as well as things that are missing.

In this blog post I am going to look at the last guideline in that list, and issues around how you specify what user an image should run as. In subsequent posts I will pull apart the other guidelines. At the end of the posts I will summarise what I believe are a better set of basic guidelines around setting up a Docker-formatted container image.

Skeleton for a Dockerfile

Following the above guidelines, the skeleton for the Dockerfile would look like:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
COPY . /app
USER app

If we build and then run this image and have it start up an interactive shell, we can validate that the command we have run is running as the user ‘app’ and that we are in the correct directory.

$ docker run -it --rm best-practices

[app@ca172749cd4f ~]$ id
uid=1001(app) gid=1001(app) groups=1001(app)

[app@ca172749cd4f ~]$ pwd

[app@ca172749cd4f ~]$ ls -las
total 24
4 drwx------ 2 app app 4096 Dec 1 03:15 .
4 drwxr-xr-x 27 root root 4096 Dec 1 03:15 ..
4 -rw-r--r-- 1 app app 18 Aug 2 16:00 .bash_logout
4 -rw-r--r-- 1 app app 193 Aug 2 16:00 .bash_profile
4 -rw-r--r-- 1 app app 231 Aug 2 16:00 .bashrc
4 -rw-r--r-- 1 root root 136 Dec 1 03:15 Dockerfile

[app@ca172749cd4f ~]$ exit

Seems simple enough, so why is this a problem?

Who do you think you can trust?

The problem is the ‘USER’ statement added to the Dockerfile. This is what declares what user the container should run as.

We can see that this was the last statement in the Dockerfile, and so this should be what user is used when the image is run. That this is the case, can be validated by inspecting the meta data of the Docker-formatted image using the ‘docker inspect’ command:

$ docker inspect --format='{{.Config.User}}' best-practices

This means that you could verify that an image satisfies the guideline that it runs as the ‘app’ user and not as the root user, before actually running it.

The problem is this doesn’t actually guarantee anything. This is because the value associated with the ‘.Config.User’ setting is a name. You cannot tell what UNIX user ID this really maps to inside of the container when run.

To illustrate the problem, consider the changed Dockerfile as follows:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN sed -i -e 's/1001/0/g' /etc/passwd
COPY . /app
USER app

Validating what ‘docker inspect’ says about what user the image will run as we still get the ‘app’ user:

$ docker inspect --format='{{.Config.User}}' best-practices

When we do run the actual image though, that isn’t the case in practice.

$ docker run -it --rm best-practices

[root@71150399f77f app]# id
uid=0(root) gid=0(root) groups=0(root)

So although the Dockerfile specified ‘USER app’ and ‘docker inspect’ also indicated that the image will run as ‘app’, the command actually ran as the root user.

This is because when using a name for ‘USER’, it still needs to be mapped to an actual UNIX user ID by the UNIX passwd file. As shown, we can remap the user name to the user ID ‘0’, meaning it still runs as the root user. We did this surreptitiously to indicate the problem of whether you can actually trust what you see. In this case the command to modify the passwd file was in the Dockerfile and in plain site, but it could also have been buried deep inside some script file or program that had been copied into the Dockerfile and then run during the building of the image.

Why does this matter?

If this is your own system you are running on for your own personal use, then you may not care. It is though a problem in a corporate setting, or if you are running a multi tenant hosting environment where you are allowing Docker-formatted images from potentially untrusted sources. In this case you want to be sure you aren’t going to run an image which actually runs as root. As we have seen, even if ‘USER’ is set in the Dockerfile to be a user other than ‘root’ it doesn’t mean it still isn’t running as root.

Verifying the user is not root

How then can we be confident that a Docker-formatted image we have been supplied isn’t going to run as root? As we have seen we obviously can’t trust ‘USER’ when it is set to a name, we have to reject any such image and not allow it to be run.

The solution is not to use a name, but the actual UNIX user ID with the ‘USER’ statement. What we therefore would require is that the Dockerfile be written as:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN sed -i -e 's/1001/0/g' /etc/passwd
COPY . /app
USER 1001

 Inspect the image now using ‘docker inspect’ and we get:

$ docker inspect --format='{{.Config.User}}' best-practices

Run the image and we get:

$ docker run -it --rm best-practices
bash-4.2$ id
uid=1001 gid=0(root) groups=0(root)
bash-4.2$ ls -las
total 36
4 drwx------ 2 1001 app 4096 Dec 1 05:10 .
4 drwxr-xr-x 28 root root 4096 Dec 1 05:12 ..
4 -rw-r--r-- 1 1001 app 18 Aug 2 16:00 .bash_logout
4 -rw-r--r-- 1 1001 app 193 Aug 2 16:00 .bash_profile
4 -rw-r--r-- 1 1001 app 231 Aug 2 16:00 .bashrc
4 -rw-r--r-- 1 root root 177 Dec 1 05:10 Dockerfile

Thus by requiring that ‘USER’ be set to a UNIX user ID, we are able to guarantee that it will run as the user it says it is. Even if the supplier of the image had still fiddled with the passwd file it wouldn’t matter, they can’t change the fact it will run as that user ID.

Recommended Guidelines

What then is a better guideline about what user a Docker-formatted container should be run? I would suggest the following.

Do not run a Docker-formatted container image as the root user. Always override in the Dockerfile what user the image will run as. This should be done by adding the ‘USER’ statement in the Dockerfile. The value of the ‘USER’ statement must be the integer UNIX user ID of the UNIX account you want any application to run as inside of the container. It should not be the user name for the UNIX account.

In addition to that guideline for the author of any Docker-formatted container image, I would also add the following guideline for anyone building a system on top of the Docker service for running images.

Where it is intended not to allow images to run as the root user, but you want to allow an image to run as the user it specifies, reject any Docker-formatted container image that you can't verify what UNIX user ID it will run as. Use ‘docker inspect’ to determine the user it should run as. Reject the image and do not run it if the user setting specified in the image meta data, is not an integer value greater than 0.

Already you will find some orchestration systems for managing containers using the Docker runtime implement this latter recommendation in certain configurations. One such example is Kubernetes and systems based around it, such as OpenShift. Because of the growth of interest in Kubernetes, especially for enterprise usage and for hosting services, adhering to the first guideline is also the first step in ensuring you will be able to deploy your images to these systems when they are set up in a secure way.

In a followup post I will look at some more aspects around what user an image should run as, whether that be the choice of the developer of the image, or whether it is a user enforced by the hosting service.

Monday, August 1, 2016

Testing out deployment of Python based Opal health care framework.

When I was working on mod_wsgi, but also in a previous job where I was working on web application performance monitoring tools, I was always after good sample Python web applications to test with. Unlike other programming languages for the web there weren’t many end user applications written in Python that you could quickly download and get running. Most of what existed out there were incomplete framework extensions which you still had to customise to get running for your own personal needs. Even if they did provide a way of starting up a skeletal application to at least see what they did, the steps to get them running were often quite complex.

One of the problems with deploying Python web applications you download are that they are often set up to be run in a very specific way with a particular hosting service or WSGI server. This fact meant you could end up spending quite a bit of time fiddling with it to get it all running in the environment you have. This made the exercise of trying to use a Python web application for testing quite frustrating at times. I can quite easily imagine that for users who might be trying to evaluate an Open Source framework extension to see if they could use it, that such difficulties in getting it running could be quite a turn off.

Where I am currently working at Red Hat as part of the OpenShift evangelist team, we are currently running a hackathon (ends 21st September) where the theme is health related applications. I have already seen that there are some Python developers out there participating in the hackathon, so I thought I might do a bit of a search around to see what I could find in the way of existing web applications or framework extensions out there for the Python programming language and test out deploying them using my warpdrive package. I actually wasn’t really expecting to find anything too interesting, but was pleasantly surprised.

One framework extension I found which I thought was quite interesting was called Opal. The Opal framework makes use of Django, along with toolkits for developing the front end such as Angular JS and Bootstrap. It was created by Open Health Care UK. The point of this blog post, as well as highlighting what looks like a quite interesting package out there that you can use, is to see how my warpdrive package stacks up when trying to deploy an arbitrary Python web application off the Internet.

Getting Opal running locally

First up is getting Opal running locally. For this Opal provides some good documentation and also a starter script to get you going. Installation and creation of an initial application I could test with was as simple as running:

pip install opal
opal startproject mynewapp

Once this was done to start up the starter application you run:

cd mynewapp
python runserver

From a browser you could then visit 'http://localhost:8000' and even login to the admin interface using a pre created user account. The latter was possible as Opal has added hooks which are automatically triggered when ‘runserver’ is used, which will set up the database and create a super user account. They have therefore optimised things for the local developer experience when using the builtin Django development server.

What now though if you wanted to deploy Opal to a production environment? They do provide a ‘Procfile’ for Heroku, but don’t provide anything which really helps out if you want to deploy to another WSGI server such as Apache/mod_wsgi or uWSGI, in a container using a local Docker service, or other PaaS environments such as OpenShift. 

It is making deployment of Python web applications easy across such different environments that my warpdrive project is targeting, so lets now look at using warpdrive to do this.

Preparing the project for warpdrive

With warpdrive already installed, the first thing we want to do is activate a new project using it. In the ‘mynewapp’ directory we run ‘warpdrive project opal’.

$ warpdrive project opal
Initializing warpdrive project 'opal'.
(warpdrive+opal) $

What this command will do is create us a new Python virtual environment just for this application and activate it. This will be an empty Python virtual environment, so next we need to install all the Python packages that the project requires.

When we originally create the project using the ‘opal startproject’ command this conveniently created for us a ‘requirements.txt’ file. This can be used with ‘pip’ to install all the packages, but we aren’t actually going to do that. This is because warpdrive also knows about ‘requirements.txt’ files and we can use it to install the required packages.

Rather than run ‘pip’ directly, we are therefore going to run ‘warpdrive build’ instead. This will not only ensure that any required Python packages are installed, but also ensures that any other framework specific build steps are also run. The output from running ‘warpdrive build’ starts out with:

 -----> Installing dependencies with pip (requirements.txt)
Collecting cryptography==1.3.2 (from -r requirements.txt (line 2))
  Using cached cryptography-1.3.2-cp27-none-macosx_10_6_intel.whl
Collecting Django==1.8.3 (from -r requirements.txt (line 3))
  Using cached Django-1.8.3-py2.py3-none-any.whl
Obtaining opal from git+git:// (from -r requirements.txt (line 18))
  Cloning git:// (to master) to /tmp/warpdrive-build.12067/opal
Installing collected packages: pycparser, cffi, pyasn1, six, idna, ipaddress, enum34, cryptography, Django, coverage, dj-database-url, gunicorn, psycopg2, static3, dj-static, django-reversion, django-axes, ffs, MarkupSafe, jinja2, letter, requests, djangorestframework, django-appconf, django-compressor, meld3, supervisor, python-dateutil, pytz, billiard, anyjson, amqp, kombu, celery, django-celery, opal
  Running develop for opal
Successfully installed Django-1.8.3 MarkupSafe-0.23 amqp-1.4.9 anyjson-0.3.3 billiard- celery-3.1.19 cffi-1.7.0 coverage-3.6 cryptography-1.3.2 dj-database-url-0.2.1 dj-static-0.0.6 django-appconf-1.0.2 django-axes-1.4.0 django-celery-3.1.17 django-compressor-1.5 django-reversion-1.8.7 djangorestframework-3.2.2 enum34-1.1.6 ffs- gunicorn-0.17.4 idna-2.1 ipaddress-1.0.16 jinja2-2.8 kombu-3.0.35 letter-0.4.1 meld3-1.0.2 opal psycopg2-2.5 pyasn1-0.1.9 pycparser-2.14 python-dateutil-2.4.2 pytz-2016.6.1 requests-2.7.0 six-1.10.0 static3-0.7.0 supervisor-3.0
Collecting mod_wsgi
Installing collected packages: mod-wsgi
Successfully installed mod-wsgi-4.5.3

One thing of note here is that ‘pip’ when run is actually trying to install Opal direct from the Git repository on GitHub. This is because the ‘requirements.txt’ file generated by ‘opal startproject’ contains:

-e git://

As far as deploying to a production environment, pulling package code direct from a Git repository, and especially from head of the master branch isn’t necessarily the best idea. We instead want to ensure that we are always using a known specific version of the package which we have tested with. To remedy this, this time we will run ‘pip’ directly, but only to uninstall the version of ‘opal’ installed so it will not cause a problems when trying to reinstall it from PyPi.

pip uninstall opal

We now edit the ‘requirements.txt’ file and replace that line with:


Worth highlighting is that this isn’t being done specially because of warpdrive. It is simply good practice to be using pinned versions of packages in a production environment so you know what you are getting. I can only imagine the ‘requirements.txt’ file is generated in this way as it makes the Opal developers job easy when testing themselves when they are working on it.

Having fixed that, we can rerun ‘warpdrive build’ and it will trigger ‘pip’ once more to ensure we have the packages we need installed, and since we removed the ‘opal’ package, it will now install the version we actually want.

Beyond installing any required Python packages, one other thing that warpdrive will do is that it will realise that the Django web framework is being used and will automatically trigger the Django ‘collectstatic’ command to collate together any static files used by the application. The next thing after package installation we therefore see in the output of ‘warpdrive build’ is:

-----> Collecting static files for Django
OSError: [Errno 2] No such file or directory: '/Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/static'

Unfortunately this fails though. The reason it fails is actually because the Django settings module for the generated Opal project contains:

# Additional locations of static files
os.path.join(PROJECT_PATH, 'static'),

With this setting, when ‘collectstatic’ is run, it expects that directory to actually exist and if it doesn’t it will fail.

This is easily fixed by creating the directory:

mkdir mynewapp/static

The directory should though have been created automatically by the ‘opal startproject’ command. That it doesn’t has already been addressed for version 0.7.1 of Opal.

After fixing this, ‘warpdrive build’ then completes successfully. It may have looked a bit messy, but that was only because we had to correct the two things related to the project that ‘opal startproject’ created for us. We could simply have left it using the ‘opal’ project from the Git repository, but felt it better to clarify what is best practice in this case.

Starting up the application

When we first started up the application we used Django’s builtin development server. This server should not though be used in a production system. Instead of the development server you should use a production grade WSGI server such as Apache/mod_wsgi, gunicorn, uWSGI or Waitress. For most use cases any of these WSGI servers will be suitable, but depending on your specific requirements you may find one more appropriate.

Setting up a project for one WSGI server even can still be a challenge in itself for many people. Trying to set up a project for more than one WSGI server so you can compare them only replicates the pain. Usually people will totally muck up the configuration of one or the other and get a totally incorrect impression of which one may actually be better.

In addition to aiming to simplify the build process, another aim of warpdrive is therefore to make it much easier to run up your WSGI application, no matter what WSGI server you want to use. This means you can get started much more quickly, but also give you the flexibility to swap between different WSGI servers.

That said, having prepared our application for warpdrive, to actually run it up and have it start accepting web requests, all we now need to do is run ‘warpdrive start’.

  -----> Configuring for deployment mode: of 'auto'
  -----> Default WSGI server type is 'mod_wsgi'
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
  -----> Running server script start-mod_wsgi
  -----> Executing server command 'mod_wsgi-express start-server --log-to-terminal --startup-log --port 8080 --application-type module --entry-point mynewapp.wsgi --callable-object application --url-alias /assets/ /Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/assets/'
Server URL : http://localhost:8080/
Server Root : /tmp/mod_wsgi-localhost:8080:502
Server Conf : /tmp/mod_wsgi-localhost:8080:502/httpd.conf
Error Log File : /dev/stderr (warn)
Startup Log File : /dev/stderr
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : en_AU.UTF-8
[Mon Aug 01 14:20:13.092722 2016] [mpm_prefork:notice] [pid 13220] AH00163: Apache/2.4.18 (Unix) mod_wsgi/4.5.3 Python/2.7.10 configured -- resuming normal operations
[Mon Aug 01 14:20:13.092995 2016] [core:notice] [pid 13220] AH00094: Command line: 'httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8080:502/httpd.conf -E /dev/stderr -D FOREGROUND'

And that is it. Our Opal application is now running.

You may be thinking at this point that using ‘runserver’ is just as easy, so what is the point, but if you look closely at the output of ‘warpdrive start’, you will see that the Django development server is not being used. Instead Apache/mod_wsgi is being used. That is, a production grade WSGI server. Not only that, you didn’t have to configure anything, all the set up and running of Apache and mod_wsgi was done for you.

Using a different WSGI server such as uWSGI is not any harder. In the case of uWSGI, all you would need to do is create the file ‘.warpdrive/server_type’ and place in it ‘uwsgi’, to override the default of using Apache/mod_wsgi. Then run ‘warpdrive build’ and once again ‘warpdrive start’ and you will instead be running with uWSGI.

  -----> Configuring for deployment mode: of 'auto'
  -----> Default WSGI server type is 'uwsgi'
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
  -----> Running server script start-uwsgi
  -----> Executing server command 'uwsgi --master --http-socket :8080 --enable-threads --threads=5 --thunder-lock --single-interpreter --die-on-term --module mynewapp.wsgi --callable application --static-map /assets/=/Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/assets/'
[uwsgi-static] added mapping for /assets/ => /Users/graham/Projects/openshift3-opal/mynewapp/mynewapp/assets/
*** Starting uWSGI (64bit) on [Mon Aug 1 14:27:02 2016] ***

In all cases, no matter which WSGI server you are using, warpdrive will worry about ensuring the minimum sane set of options are provided to the WSGI server as well as any required for the specific WSGI application. In this case warpdrive even handled the task of making sure the WSGI server knew how to host the static files the application needs.

Initialising an application database

Our Opal application is again running and we can access it via the browser from 'http://localhost:8080/'. Do so though and we encounter a new problem though.

Exception Type: OperationalError
Exception Value: no such table: axes_accessattempt

This gets back to that magic that was being done when ‘runserver’ was being used. Specifically, the ‘runserver’ command had been set up to also automatically ensure that the database being used was initialised and that an initial account created.

Doing that for a development system is fine, but you would have to be careful about automating that in a production system. For starters, although in a development system you can use a file based database such as SQLite, in production you are more likely going to be using a database such as MySQL or PostgreSQL. These will be handling your real data and so you have to be much more careful in what you do with those databases.

When using Django, which Opal is based on, it provides two management commands for initialising a database and creating accounts. These are ‘migrate’ and ‘createsuperuser’. The ‘migrate’ command actually serves two purposes. It can be used to initialise the initial database, but also perform database migrations when the database model used by the application code changes.

These are slightly magic steps which you need to know about how Django works to know how to run. When they were automatically triggered by ‘runserver’ you didn’t have to know how to run them as that knowledge was coded into the scripts triggered by ‘runserver’.

As codifying such steps is beneficial from the stand point of ensuring that such steps are captured and always done the same way, warpdrive provides a mechanism called action hooks for recording what these steps are. You can then get warpdrive to run them for you and you don’t have to know the details. You can embed in the action as much magic as you need to, including steps like ensuring that your database is actually running before attempting anything, or allowing details of accounts to create to be supplied through environment variables or configuration files.

As an example, lets create our first action hook. This we will save away in the file ‘.warpdrive/action_hooks/setup’.

echo " -----> Running Django database migration"
python migrate
if [ x"$DJANGO_ADMIN_USERNAME" != x"" ]; then
    echo " -----> Creating predefined Django super user"
    (cat - | python shell) << !
from django.contrib.auth.models import User;
if (tty > /dev/null 2>&1); then
echo " -----> Running Django super user creation"
python createsuperuser

This captures the steps we need to initialise the database and create an initial account. For the account creation, we can either supply the details via environment variables, or if we are running in an interactive shell, it will prompt us. Having created this action hook we can now run ‘warpdrive setup’.

  -----> Running .warpdrive/action_hooks/setup
-----> Running Django database migration
Operations to perform:
Synchronize unmigrated apps: search, staticfiles, axes, messages, compressor, rest_framework
Apply all migrations: sessions, admin, opal, sites, auth, reversion, contenttypes, mynewapp
Synchronizing apps without migrations:
Creating tables...
Creating table axes_accessattempt
Creating table axes_accesslog
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying sites.0001_initial... OK
-----> Running Django super user creation
Username (leave blank to use 'graham'):
Email address:
Password (again):
Superuser created successfully.

Running ‘warpdrive start’ once again we find the application is now all working fine and we can log in with the account we created. 

The contents of the ‘setup’ script is typical here of what is required for database initialisation when using Django. The other set of actions we want to capture for Django is what needs to be done when migrating the database after database model changes. These we can capture in the file ‘.warpdrive/action_hooks/migrate’.

echo " -----> Running Django database migration"
python migrate

Why it is better to capture these commands as action hooks and have warpdrive execute them for you, is that the commands are now a part of your application code. You don’t need to go look up some documentation to remember what the steps are. All you need to remember is the commands ‘warpdrive setup’ and ‘warpdrive migrate’.

Another important reason is that if there are any special environment variables that need to be set to replicate the actual environment when your web application is run, warpdrive will also worry about setting those as well. This means you wouldn’t need to remember to set some special value for the ‘DJANGO_SETTINGS_MODULE' environment variable in order to run the Django management commands directly. The warpdrive command will know what is required and set it up for you based on what you have captured about that in your application code.

Moving to a production environment

Using ‘warpdrive’ in our local environment has allowed us to more easily use a production grade WSGI server during development. Using the same WSGI server as we will use in production means we are more likely to pick up problems which will not show up using the development server.

The action hooks feature of warpdrive has also resulted us in capturing those important steps we need to run to initialise any database and later perform database migrations when we make changes to our database model.

That is a good start, but what now if we want to run our Opal application in a production environment?

The first example of how we might want to do that is to use Docker. For that though we first need too create a Docker image which contains our application along with any WSGI server and configuration needed to startup the application.

This step is where people often waste quite a lot of time. Developers can’t resist new toys to play with and so they feel it is imperative that they learn everything about this new Docker tool, throwing away any wisdom they may have accumulated about best practices over time and start from scratch, building up their own special Docker image piece by piece.

More often than not this results in a poorly constructed Docker image that doesn’t follow best practices and which can well be insecure, running as root and requiring it be run in a way that could easily lead to being able to break into your wider systems if someone can compromise your web application.

With warpdrive there is a much better way of moving to Docker. That is to have warpdrive build the Docker image for you. You don’t need to know anything about creating Docker images as warpdrive will build up the image ensuring that best practices are being used.

To package our Opal application up into a Docker image, all we need to do is run the ‘warpdrive image’ command.

(warpdrive+opal) $ warpdrive image opal
I0801 15:35:29.321041 14900 install.go:251] Using "assemble" installed from "image:///opt/app-root/s2i/bin/assemble"
I0801 15:35:29.321223 14900 install.go:251] Using "run" installed from "image:///opt/app-root/s2i/bin/run"
I0801 15:35:29.321280 14900 install.go:251] Using "save-artifacts" installed from "image:///opt/app-root/s2i/bin/save-artifacts"
  ---> Installing application source
  ---> Building application from source
  -----> Installing dependencies with pip (requirements.txt)
Collecting cryptography==1.3.2 (from -r requirements.txt (line 2))
Downloading cryptography-1.3.2.tar.gz (383kB)
Collecting Django==1.8.3 (from -r requirements.txt (line 3))
Downloading Django-1.8.3-py2.py3-none-any.whl (6.2MB)
-----> Collecting static files for Django
78 static files copied to '/opt/app-root/src/mynewapp/assets', 465 unmodified.
---> Fix permissions on application source

This should look familiar to you as in building the Docker image it is using the same ‘warpdrive build’ command that you used in your local environment. This is being done within a Docker base image which has already been set up with Python and warpdrive.

By using the same tooling, in the form of warpdrive, in your local environment as well as in constructing the Docker image, you have a better guarantee that things are being set up in the same way and will also run in the same way. This removes the disparity that usually exists between working in a local environment and what you have in your production environment.

The final result of that ‘warpdrive image’ command is that you now have a Docker image named ‘opal’ which we can be run using ‘docker run’.

(warpdrive+opal) $ docker run --rm -p 8080:8080 opal
---> Executing the start up script
-----> Configuring for deployment mode: of 'auto'
-----> Default WSGI server type is 'mod_wsgi'
Python 2.7.12 (default, Jul 29 2016, 00:52:26)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
  -----> Running server script start-mod_wsgi
  -----> Executing server command 'mod_wsgi-express start-server --log-to-terminal --startup-log --port 8080 --application-type module --entry-point mynewapp.wsgi --callable-object application --url-alias /assets/ /opt/app-root/src/mynewapp/assets/'
Server URL : http://localhost:8080/
Server Root : /tmp/mod_wsgi-localhost:8080:1001
Server Conf : /tmp/mod_wsgi-localhost:8080:1001/httpd.conf
Error Log File : /dev/stderr (warn)
Startup Log File : /dev/stderr
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : en_US.UTF-8
[Mon Aug 01 05:49:45.774485 2016] [mpm_event:notice] [pid 20:tid 140425572333312] AH00489: Apache/2.4.23 (Unix) mod_wsgi/4.5.3 Python/2.7.12 configured -- resuming normal operations
[Mon Aug 01 05:49:45.774622 2016] [core:notice] [pid 20:tid 140425572333312] AH00094: Command line: 'httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8080:1001/httpd.conf -E /dev/stderr -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

Like with how ‘warpdrive build’ was used in constructing the Docker image, the ‘warpdrive start’ command is also used in the final container when run.

We are still only using the file system database SQLite, which will not survive the life of the container, at this point, and we also need to initialise that database, but we can again use the ‘warpdrive setup’ command.

$ docker exec -it berserk_galileo warpdrive setup
-----> Running .warpdrive/action_hooks/setup
-----> Running Django database migration
Operations to perform:
Synchronize unmigrated apps: compressor, staticfiles, search, messages, rest_framework, axes
Apply all migrations: sessions, contenttypes, admin, mynewapp, sites, reversion, auth, opal
Synchronizing apps without migrations:
Creating tables...
Creating table axes_accessattempt
Creating table axes_accesslog
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
-----> Running Django super user creation
Username (leave blank to use 'default'): graham
Email address:
Password (again):
Superuser created successfully.

This is where the benefit of having captured all those steps to initialise the database in an action hook comes into play. You only need to know the one command and not all the individual commands.

Making an application production ready

As you can see using warpdrive can certainly help to simplify getting a Python web application running and getting it into production using a container runtime such as Docker. Part of the benefits of using warpdrive are that it handles the WSGI server for you, but also features like action hooks, which help to ensure you capture important key steps around how to setup your application.

There is still more to getting an application production ready than just this though. Especially when using containers, because the running container is ephemeral and so any local data is lost when the container stops, it is important to use an external database or persistent storage. Work also needs to be done around how you configure your application as well as log information from it.

In followup posts I will start to delve into these issues and how warpdrive can help you configure your application for the target environment. I will also go into detail about how warpdrive can be used with PaaS offerings such as OpenShift.