Comments on Graham Dumpleton: Load spikes and excessive memory usage in mod_pyth...

@Ramdas, there is no single ideal configuration, i...

2009-07-08T17:24:09.278+10:00

@Ramdas, there is no single ideal configuration, it depends on your specific application and also where hosting multiple sites the amount of traffic going to specific sites and then memory used by each.

If you want to have a discussion about it, use the mod_wsgi list on Google Groups to explain your problems/requirements.

Graham, Good points made. However your para on mod...

2009-07-08T14:58:54.707+10:00

Graham,
Good points made. However your para on modwsgi in daemon mode vs embedded mode wasn't quite clear.

My question if its a dedicated server and you are running multiple sites what should be the ideal configuration of modwsgi. Say I have 8 GB RAM.
Is there aplace which teaches me running modwsgi in dameon mode

The thread stack size issue is described in: http...

2009-05-27T19:22:03.699+10:00

The thread stack size issue is described in:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Memory_Constrained_VPS_Systems

Very important on at least an openVZ VPS is to red...

2009-05-27T18:55:10.437+10:00

Very important on at least an openVZ VPS is to reduce the per thread stack size. It is 8MB by default on linux. There is ThreadStackSize directive for apache, but I use ulimit -s 256 in the apache startup script. If you use mod_wsgi in daemon mode, it has a stack-size option you can use.

Graham, thanks for taking the time to write this u...

2009-03-27T02:27:00.000+11:00

Graham, thanks for taking the time to write this up. It was very useful.

@Graham Thanks a bunch!

I find it quite unf...

2009-03-20T22:19:00.000+11:00

@Graham Thanks a bunch!

I find it quite unfortunate that this 'little detail' (I mean the python webapp deployment model) is so muddy in the Python world and I think you are doing an excellent job clarifying most of its aspects.

I am preparing a blog post myself summarizing my findings (which will most probably link heavily to your posts :-)) and I hope you'll not mind if I'll

@alex

If the number of processes you config...

2009-03-20T16:38:00.000+11:00

@alex

If the number of processes you configure fastcgi or mod_wsgi daemon mode to use for a specific application is the same as the maximum number of Apache child server processes, whether using worker or prefork MPM, then yes you would expect overall memory usage to thus be roughly the same for fat Python web applications.

For a very small Python web application, then obviously

I've been documenting myself about the various alt...

2009-03-20T16:11:00.000+11:00

I've been documenting myself about the various alternatives available for deploying python webapps for quite a while and I have found your post very helpful in confirming my thoughts (I'm not a very experienced Pythonist, but spent most of the last decade on the Java platform).

There is still one thing that I'm not 100% sure I do understand. Leaving aside the main difference between

@kmike

You don't say what options you are u...

2009-03-14T16:35:00.000+11:00

@kmike

You don't say what options you are using to ab/siege in respect to number of concurrent connections. Not obvious if you have eliminated preloading overheads or have ensure that aren't counting into your results. So, hard to comment. Maybe the mod_wsgi list would be a better place to discuss this as am interested in this so can try and develop some sort of rule of thumb that can be

With no sql queries the results are basically the ...

2009-03-14T11:36:00.000+11:00

With no sql queries the results are basically the same.

I see, I was wrong about GIL and 3d party extensio...

2009-03-14T10:32:00.000+11:00

I see, I was wrong about GIL and 3d party extensions issuses.

I've done some apache+mod_wsgi (daemon mode) benchmarks with "ab" and "siege" utilities and the results seems confusing to me.

The test page was django powered and have to execute ~15 SQL queries (mysql) to render.

The configs was:

1) WSGIDaemonProcess site-1 user=user-1 group=user-1 threads=25
<

Obviously, instead of 'So, once things have starte...

2009-03-14T07:43:00.000+11:00

Obviously, instead of 'So, once things have started up, you would be affected.', meant to say 'So, once things have started up, you would NOT be affected.'

@kmike

If a database adapter or other third...

2009-03-14T07:41:00.000+11:00

@kmike

If a database adapter or other third party C extension module doesn't unlock the GIL before going into a blocking operation then that module is fundamentally flawed and it should be reported as a bug in that module. None of the major database adapters available have this problem as far as I am aware, so please be more specific about which extension modules you believe are broken.<

I'm sorry to disagree with some points from your p...

2009-03-14T00:06:00.000+11:00

I'm sorry to disagree with some points from your post.

Prefork is much better for Python because of GIL (when your scripts use database). Database access (and some other Python modules) causes threads to lock so only 1 thread in process is active. This causes major performance penalty - all threads in process are waiting while SQL query is executing. If you use prefork mpm GIL won't

@aarone

In respect of stability of worker M...

2009-03-13T13:48:00.000+11:00

@aarone

In respect of stability of worker MPM, are you talking about the worker MPM implementation itself within Apache, or the ability of mod_python and/or a specific Python web application to work properly in a multithreaded environment?

Personally I have never heard of anyone having an issue with stability of Apache worker MPM itself. Also know of no multithreading issues in

If mod_php is able to share libraries between proc...

2009-03-13T12:09:00.000+11:00

If mod_php is able to share libraries between processes with prefork, why can't mod_python?

Is there a problem with PHP's model of pre-loading and sharing libraries (e.g. security)?

If mod_php chose a "stupid" solution that seems to work well with shared nothing, how is the mod_python solution any better?

My guess is that mod_python didn't do it that way out of any

This is excellent post. Thanks.

Regards,
2009-03-10T17:51:00.000+11:00
This is excellent post. Thanks.

Regards,
Distributed Enterprise Application

mod_wsgi is light years better than mod_fcgi.

2009-03-10T05:33:00.000+11:00

mod_wsgi is light years better than mod_fcgi.

You can also use mod_fcgi. At this point it might ...

2009-03-10T00:13:00.000+11:00

You can also use mod_fcgi. At this point it might make sense to stop using Apache and use something like lighttpd or nginx.

Comments on Graham Dumpleton: Load spikes and excessive memory usage in mod_pyth...

@Ramdas, there is no single ideal configuration, i...

Graham, Good points made. However your para on mod...

The thread stack size issue is described in: http...

Very important on at least an openVZ VPS is to red...

Graham, thanks for taking the time to write this u...

@Graham Thanks a bunch!I find it quite unf...

@alexIf the number of processes you config...

I've been documenting myself about the various alt...

@kmikeYou don't say what options you are u...

With no sql queries the results are basically the ...

I see, I was wrong about GIL and 3d party extensio...

Obviously, instead of 'So, once things have starte...

@kmikeIf a database adapter or other third...

I'm sorry to disagree with some points from your p...

@aaroneIn respect of stability of worker M...

If mod_php is able to share libraries between proc...

This is excellent post. Thanks.Regards, 2009-03-10T17:51:00.000+11:00 This is excellent post. Thanks.Regards,Distributed Enterprise Application

mod_wsgi is light years better than mod_fcgi.

You can also use mod_fcgi. At this point it might ...

@Graham Thanks a bunch!

I find it quite unf...

@alex

If the number of processes you config...

@kmike

You don't say what options you are u...

@kmike

If a database adapter or other third...

@aarone

In respect of stability of worker M...

This is excellent post. Thanks.

Regards,
2009-03-10T17:51:00.000+11:00
This is excellent post. Thanks.

Regards,
Distributed Enterprise Application