Integrating mod_wsgi-express as a Django admin command.

To followup on my first post introducing mod_wsgi-express, I posted about how to use mod_wsgi-express with Django. This provided us with a workable solution but did result in us needing to duplicate some information which was available within the Django settings file on the command line to mod_wsgi-express.

The next step therefore was to avoid this requirement by integrating mod_wsgi-express into the Django site itself so that it can be executed as a Django management command. By integrating mod_wsgi-express in this way it can directly interrogate the Django settings module for your Django project to obtain the information it wants.

In short, instead of having to run:

mod_wsgi-express start-server --url-alias /static static --application-type module mysite.wsgi

our goal is to be able to run:

python manage.py runmodwsgi

Updating installed applications

The actual integration of mod_wsgi-express into the Django project is actually a quite simple matter. This is because the mod_wsgi package implements the Django application abstraction. We therefore need only list the appropriate mod_wsgi sub package in ‘INSTALLED_APPS’ in the Django settings file.

INSTALLED_APPS = (
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'mod_wsgi.server',
)

In particular we have added ‘mod_wsgi.server’ to ‘INSTALLED_APPS’.

The result of doing this is to extend the Django project such that the ‘runmodwsgi’ command is now available as a management command when running ‘python manage.py’.

When ‘runmodwsgi’ is now run a number of pieces of information are extracted from the Django settings file to allow it to run correctly.

The first is that the value of ‘BASE_DIR’ from the Django settings file is used to determine the working directory for mod_wsgi-express when it is run. This directory is automatically added into the Python module search path so that the Python package corresponding to the site can be found, no matter where ‘manage.py’ was run from.

Next is that the ‘WSGI_APPLICATION’ setting is used to determine the actual true location of the WSGI application entry point.

In our initial use of mod_wsgi-express we relied on the WSGI application entry point being stored in the ‘wsgi.py’ module file within the Python package for the site. Further, we relied on the WSGI application entry point still being called ‘application’, which so happens to coincide with the default that mod_wsgi-express looks for.

In practice neither of these need be true as a user may decide to ignore the ‘wsgi.py’ generated by the ‘startproject’ management command and either use a different module within the package or change the name of the WSGI application entry point function. If a user had done this and were intending to still use the Django development server, it would have been necessary to update the ‘WSGI_APPLICATION’ setting. As the Django development server uses ‘WSGI_APPLICATION’ so we do with the ‘runmodwsgi’ management command.

Finally, we use the values of ‘STATIC_URL’ and ‘STATIC_ROOT’ to determine where any static media files will exist, subsequent to ‘collectstatic’ having being run, and at what sub URL they should be made available.

With this information being automatically picked up, we can now say:

python manage.py runmodwsgi

So we have achieved our goal, but there is at least one more change that you might make to improve the experience when using mod_wsgi-express with the Django site. Or at least when using it in a development environment.

Logging of Python exceptions

When using Django, by default the details of any exceptions occurring in your code when handling a request are not logged in any way. To know about any exceptions occurring you need to update the Django settings file and configure one of a number of different possible ways for capturing them. In a production environment, which you would use depends on the strategy you are using for monitoring your applications.

The two primary mechanisms often used are to configure Django to send you an email for each exception which occurred, or to simply log the details in the error log for your WSGI server. It is this latter mechanism which we are going to set up here. For this we are going to add to the Django settings file:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'),
        },
    },
}

With this in place the details of any exceptions will be logged to the Apache error log for mod_wsgi-express. The path to the error log will be displayed to the terminal when running mod_wsgi-express so you know where it is.

As the Apache error log is a distinct file, when using mod_wsgi-express during development, the need to consult the separate error log file may be inconvenient. In this case one can instead tell mod_wsgi-express to log all information to the terminal instead.

python manage.py runmodwsgi --log-to-terminal

If you are using mod_wsgi-express in a development environment but are using a different WSGI server in production, even if a normal Apache/mod_wsgi installation, having the exceptions logged to the error log may not always be appropriate. If this is the case and you would only want the exception details logged to the error log when using mod_wsgi-express, you can check for the existence of an environment variable which tells you if mod_wsgi-express is being used.

if os.environ.get('MOD_WSGI_EXPRESS'):
    LOGGING = {
        …
    } 

Use as a development server

Now I keep mentioning things you might want to do in a development environment, but mod_wsgi-express isn’t just about development environments. As Apache/mod_wsgi is used under the hood, it is more than suitable for use in production environments. As is always the case when deploying to a production server, you want to ensure that you do tune the server configuration properly for that environment.

By default mod_wsgi-express will create a single process to run your Django site in, with multithreading being used to handle concurrent requests. The default number of request handler threads used by the process is 5. So when moving to production consideration should be given to adjusting both the number of processes as well as the number of threads.

As well as tuning the capacity of the server in this way, there are various other considerations that need to be made. One of the primary things is that mod_wsgi-express when run directly as shown, will actually run the server in the foreground and will not run it as a daemon. This is so that it is convenient to use it in a development environment, where shutting down the server is as simple as doing CTRL-C in the terminal window where you ran it.

For production you wouldn’t want to use mod_wsgi-express in exactly this way. Instead there is a way of having mod_wsgi-express generate the configuration and and a set of management scripts but not actually run the server. This captures all the information about the site and then later one can use the management script to start/stop the site as necessary from any operating system start up scripts or process management system.

Anyway, it is not my aim to talk about moving mod_wsgi-express into production at this point, instead I want to focus more about using it in a development environment and why it can be a better solution for that than the builtin Django development server.

The first reason as noted above is that mod_wsgi-express runs in a multithreaded configuration. In contrast when you use the builtin development server it runs with a single process and a single thread. This means it is too easy to develop code which works fine in the Django development server but then fails in production when using a multithreaded WSGI server.

By using a production capable WSGI server such as mod_wsgi-express during development, where both multi process and multithreaded configurations can be setup, you are able to better replicate the environment you will expect to run under in production. So although mod_wsgi-express does use multithreading by default, to encourage the creation of thread safe code, if you truly did want to avoid multithreading and wanted to use single threaded processes, you could use:

python manage.py runmodwsgi —processes 3 --threads 1

The benefits of using mod_wsgi-express also extend to the serving up of static media files as these are now also being handled by a proper web server. You can therefore get a better idea of how your web application is going to perform in production because you can do load testing against mod_wsgi-express and know that it isn’t going to be too different to what you would expect in a production environment, ignoring of course differences in hardware or operating system.

Reloading of source code

You might be saying at this point though that one of the benefits of using the Django development server is that it offers automatic source code reloading. Well, mod_wsgi-express has that as well.

python manage.py runmodwsgi --reload-on-changes

With the ‘—reload-on-changes’ option to mod_wsgi-express, any code files associated with modules that have been loaded into the running Python web application will be monitored for changes. If any of those files on disk change in between the time it was loaded and when the check was made, then the web application process(es) will be automatically shutdown and restarted. In this way, when the next request is made it will see the more recent code changes.

Do note though that this doesn’t extend to any changes made to the static media files in their original locations. This is because the running of ‘collectstatic’ resulted in the original files being copied into the common directory so they could be served. As a result, for changes to CSS stylesheets, Javascript and images to be picked up, you will need to run ‘collectstatic’ again. You will not however have to restart mod_wsgi-express, as once ‘collectstatic’ is run, it should then pick up the latest versions.

Even with just the ability to automatically reload on code changes we have already covered what the Django development server is principally relied upon for, but mod_wsgi-express implements a whole bunch of other features as well which go beyond what the builtin development server provides.

These additional features are part of a special debug mode and include builtin support for debugging exceptions using pdb, profiling of code, code coverage and recording of requests, including headers and content. I will cover debug mode of mod_wsgi-express in the next blog post.