Use of threading in mod_wsgi daemon mode.

Every so often I get asked why when examining a mod_wsgi daemon process do there appear to be more threads running than what have been configured to handle requests. Since I can never be bothered trying to find what I wrote the previous time, I end up writing the explanation from scratch each time. Figured it may be time to simply throw it up here on my blog so I can at least refer to it here.

Consider therefore the Apache/mod_wsgi configuration of:

WSGIDaemonProcess mysite processes=3 threads=2 display-name=mod_wsgi
WSGIProcessGroup mysite
WSGIScriptAlias / /some/path/wsgi.py

What this results in is mod_wsgi creating a set of three processes distinct from the main Apache child worker process. Multiple instances of the WSGI application will then be run, one in each process. The Apache child worker processes will automatically proxy the requests, with a request being picked up by one of the daemon processes and handed off to the WSGI application to handle. With the number of threads being specified as two, the most concurrent requests each process can handle is two. With a total of three processes, that is a total of six concurrent requests able to be handled across the whole WSGI application.

If we were to now look at the resulting process tree using something like htop tree view, we would see:

20179 www-data 20 0 147M 76408 5680 S 28.0 0.9 10:17.85 │ ├─ mod-wsgi -k start
20240 www-data 20 0 147M 76408 5680 S  1.0 0.9  0:14.99 │ │ ├─ mod-wsgi -k start
20215 www-data 20 0 147M 76408 5680 S 12.0 0.9  5:05.16 │ │ ├─ mod-wsgi -k start
20214 www-data 20 0 147M 76408 5680 S 14.0 0.9  4:53.99 │ │ ├─ mod-wsgi -k start
20213 www-data 20 0 147M 76408 5680 S  0.0 0.9  0:00.63 │ │ ├─ mod-wsgi -k start
20212 www-data 20 0 147M 76408 5680 S  0.0 0.9  0:00.00 │ │ └─ mod-wsgi -k start

20178 www-data 20 0 138M 67680 5212 S 52.0 0.8 11:01.62 │ ├─ mod-wsgi -k start
20241 www-data 20 0 138M 67680 5212 S  0.0 0.8  0:15.45 │ │ ├─ mod-wsgi -k start
20230 www-data 20 0 138M 67680 5212 S 15.0 0.8  5:17.81 │ │ ├─ mod-wsgi -k start
20229 www-data 20 0 138M 67680 5212 S 35.0 0.8  5:24.63 │ │ ├─ mod-wsgi -k start
20228 www-data 20 0 138M 67680 5212 S  0.0 0.8  0:00.71 │ │ ├─ mod-wsgi -k start
20227 www-data 20 0 138M 67680 5212 S  0.0 0.8  0:00.00 │ │ └─ mod-wsgi -k start

20177 www-data 20 0 137M 67764 5428 S  7.0 0.8 10:47.27 │ ├─ mod-wsgi -k start
20207 www-data 20 0 137M 67764 5428 S  0.0 0.8  0:15.18 │ │ ├─ mod-wsgi -k start
20206 www-data 20 0 137M 67764 5428 S  7.0 0.8  5:16.82 │ │ ├─ mod-wsgi -k start
20205 www-data 20 0 137M 67764 5428 S  0.0 0.8  5:11.55 │ │ ├─ mod-wsgi -k start
20204 www-data 20 0 137M 67764 5428 S  0.0 0.8  0:00.69 │ │ ├─ mod-wsgi -k start
20203 www-data 20 0 137M 67764 5428 S  0.0 0.8  0:00.00 │ │ └─ mod-wsgi -k start

The question is, since we have configured the daemon process group to only have 2 threads per process, why are we seeing a total of 5 threads in each process?

The answer is that for a configuration of:

WSGIDaemonProcess mysite threads=num ...

there will be num+3 threads, where 'num' is the number of request threads indicated by the 'threads' option to the WSGIDaemonProcess directive. If no 'threads' option was specified, then the number of request threads will be 15.

What are these three extra threads then? They are as follows:

1. The main thread which was left running after the daemon process forked from Apache. It is from this thread that the requests threads are created initially. It will also create the other two additional threads, the purpose of which is described below. After it has done this, this main thread becomes a caretaker for the whole process. It will wait on a special socketpair, which a signal handler will write a character to as a flag that the process should shutdown. In other words, this main thread just sits there and stops the process from exiting until told to.

2. The second thread is a monitoring thread. What it does is manage things like the process activity timeout and shutdown timeout. If either of those timeouts occur it will send a signal to the same process (ie., itself), to trigger shutdown of the process.

3. The third thread is yet another monitoring thread, but one which specifically detects whether the whole Python interpreter itself gets into a complete deadlock and stops doing anything. If this is detected it will again send a signal to the same process to trigger a shutdown.

So the additional threads are to manage process shutdown and ensure the process is still alive and doing stuff.

Now under normal circumstances there will be one further additional thread created, but this is a transient thread which is only created at the time that the main thread has detected that the process is due to shutdown. This thread is what is called the reaper thread. All it will do is sleep for a specified period and if the process is still running at that point, it will forcibly kill the process.

The reaper thread is needed because the main thread will provide a short grace period to allow requests to complete and then destroy the Python interpreter, including triggering of any 'atexit' registered callbacks in each sub interpreter context. Because pending requests and destruction of the interpreters could take an unknown amount of time, or even hang, the reaper thread ensures the process is still killed if shutdown takes longer than allowed.

And there we have it, all intentional and nothing to worry about.