Thursday, October 21, 2010

Is PEP 3333 the final solution for WSGI on Python 3?

While I was off on holidays PJE took PEP 333 for WSGI and did minor updates to it to accommodate Python 3. The result was PEP 3333. The response to this on the Python WEB-SIG was pretty muted, with the only significant person giving open support that I could find was Guido himself. Guido's comment was:
This is a very laudable initiative and I approve of the changes
Is this now the end of the matter and are people going to accept this as the final solution? Or are we still going to have a problem with people offering alternate proposals for a standard web server interface for web applications under Python 3.X?

As far as I can see PEP 444 is more or less still on the table as a future alternative, albeit that it may need some more work.

So, what do people think? Is PEP 3333 what people are going to use, or is it going to be still born and people are going to demand something else, be that PEP 444 or otherwise?

Or is it all academic now given that PEP 3333 is using the WSGI name and so there can be no alternative, at least under the WSGI name, with anything else having to be called something completely different anyway?

I would really like to hear from the core developers from the major web frameworks and what you expect you will now do based on PEP 3333 being created. Based on that feedback, I will work out what to do with Apache/mod_wsgi in respect of bringing it into alignment with PEP 3333.

Ultimately I guess if PEP 3333 doesn't get used, just means that Apache/mod_wsgi will principally be for Python 2.X and a Python 3.X solution for Apache based on alternate specification will be a distinct package, maybe done by me, but possibly left up to someone else.

16 comments:

Daniel said...

Your post is not quite clear if you solicit answers from "significant" people or are throwing the question out in the wind...

I'll interpret that as the latter and offer a simple view on from the point of "small WSGI app coder." I'm sure you have plenty of me-likes chiming in all the time, still...


3333 Feels like a simple, proper fix for 333's most glaring issues - bottomless wsgi.input, embrace of some HTTP/1.1 realities.

The fact that 3333 tippy-toes around unicode issue in the most benign (for app developr) way is almost secondary, but as an WSGI app writer I am much more pleased with 3333's approach to that than 444's. 444 is almost the same in substance, but the verbiage evokes a type Nazi in me.
As a user I would like not to care about the Latin-1 nonesense and have all environ, header etc (except request body and response body) be "Unicode" and have the server jump the extra hoops to decode / accommodate / raise exceptions. 3333 sifts the string type requirements down to manageable, understandable 2-3 sentences.

444 and 3333 serve different goals. 3333 just patches specific, most egregious goals and is long overdue. 444 (seeing "async" in there) embarks on philosophical discussion of "where do we go from here?" And, i don't see much future forest for the trees in 444 at this time.
444 can be stuffed with more stuff, later. Example: "What do you do with Trailer headers?" (Not an important issue, but, I say push them into the same "environ" pointer, after server already passed it to the app. CherryPy has an odd-ball method on wsgi.input obj for getting Trailers after reading is done.)

One thing that seriously upsets me in 3333. wsgi.version is still (1.0) per spec. It needs to change, to something like (1,1) as handling of wsgi.input can be so fundamentally different and simple (and possible now) per 3333 under HTTP/1.1, Chunked.

I'd switch to 3333-compatible server on the spot. CherryPy 3.2, WSGI server is essentially that already. Let's get 3333 advertise itself with (1,1) and roll with it.

Graham Dumpleton said...

Daniel, am after responses from both. The key players in the major web frameworks, but also anyone else who has read enough about all the proposals to understand what the issues are, such as you appear to have done.

For the record, I have a preference for seeing it as 1.1 for wsgi.version as well.

jdhardy said...

My impression (as the author of a WSGI gateway for IronPython) is that PEP 3333 is 'good enough' for Unicode support (and everything else, but Unicode is important to IronPython), although my opinion may change once I actually sit down to implement it.

Rewriting applications just to get proper unicode support is a no-go. I think PEP 444 is ultimately needed as well, but I'll settle for a fixed PEP 333 at this point.

Nicholas said...

PEP 3333 will allow us to let go of Python 2 and see a consolidation of energy behind Python 3.

PEP 444 requires experiment, proving, and rationale. uWSGI has stepped up with a server-side implementation, but we have not seen any framework implementation behind it.

3333 is the pragmatic choice, 444 is bones for bike shedders. Let's move on. Nothing excludes future development of web3.

Graham Dumpleton said...

For the record, discussion about this is on Python WEB-SIG at 'http://groups.google.com/group/python-web-sig/browse_frm/thread/2746491c020e4e67'. The feeling expressed is that PEP 3333 is it for now. Anything else more or less deferred for time being.

Daniel said...

Scanning the conversations about PEP-3333 on Web SIG couldn't help by wonder if the effort will be still-born.

The way I see it, the v3-fication of WSGI standard is mostly just a vehicle to fix WSGI 1.0 bugs that mire both, Python 2.x and 3.x.
There was really only one thing wrong with PRP 333 - wsgi.input was bottomless, which is fatal in HTTP/1.1 Chunked world.

I see conversations about making PEP 3333 somehow Python 3.x-specific because of transcoding nuances for CGI deployment scenario. Stuff like that is "throwing out the baby with the bath water."

I fear that minutia about os.environ transcoding and other 3.x specific fluff will hold back the real bug fixes due for WSGI API. I'm afraid the desire to stay "compatible" with WSGI 1.0 which, likely manifested itself in wsgi.verion = 1.0 (vs. 1.1) in PEP 3333 will turn the WSGI App implementation into guessing and server-sensing nightmare.
Yet, I am afraid to raise any concerns anywhere because talking about these things historically only prolonged the fixes to WSGI standard.

Demn, as a developer, I am liking WSGI world less and less...

Graham Dumpleton said...

Daniel. It isn't just the WEB-SIG discussions, it when you see posts like this where it is obvious that people are still hanging out for PEP 444. This suggests that they will not go Python 3.X with PEP 3333 as a base.

GothAlice said...

I've implemented a ultra-performance (6Krsecs at C10K, 10Krsecs at much lower concurrency) HTTP/1.1 (mostly, chunked encoding is a WIP) server that is (in theory, needs testing) compatible with 2.6+ and 3.1+ that implements PEP 444 (Web3) with some modifications. I'll be developing it and related tools against PEP 444, which seems to be more clearly defined -- no support for 2.5 and below, for example, makes cross-version compatibility much more of a reality. The server itself is alpha, and I never expected to be able to produce the performance I have. (I just wanted to see what it was like to write one, possibly make something useful for debugging.)

I've posted to the Web-SIG mailing list my ideas on PEP-444, and my server implements them.

Server Overview:
http://web-core.org/marrow/server/http-design/

Server Source:
https://github.com/pulp/marrow.server.http

Mailing list message:
http://web-core.org/2.0/pep-0444/

Graham Dumpleton said...

GothAlice, your trying to push forward PEP 444 at this time is just making things uncertain again when they were starting to clarify that PEP 3333 would be the way forward at this point. Please just wait until the PEP 3333 issue has been settled, or do what you are doing well away from the WEB-SIG and not refer to it as in any way connected to WSGI. You may find the latter necessary anyway, as the WEB-SIG is likely to ignore you at this point and you aren't going to get any support from anyone.

Graham Dumpleton said...

Further discussion discussion around alternatives at http://groups.google.com/group/python-web-sig/browse_frm/thread/e5deeb88acf23b46

It is that these discussions keep coming up that concerns me as far as whether the commitment to PEP 3333 will actually be there.

Right now there is still no definitive public statement as to what course will be taken out of the WEB-SIG or wider Python web community. :-(

GothAlice said...

Truth be told PEP 333 (WSGI) has been "just enough" for the vast majority of WSGI developers. There are glaring issues, but as mentioned (see: CherryPy) are already addressed by existing server solutions. PEP 3333 while worthy as a 1.1 extension to WSGI 1 is inherently limited by its vagueness on certain critical issues for portability to Python 3. It really does feel like a .1 release.

PEP 444, for which I have been granted control, seems to be a far more Py3K-style solution by explicitly breaking backwards compatibility (beyond 2.6). After a small amount of discussion on the Web-SIG, PEP 444 has been re-named WSGI 2 (from "web3") and my draft includes that and a few other changes/additions. One notable thing removed from the draft I currently host is async support, which is unfortunate, but more experimentation needs to be done before I can define clearly how async can be supported by WSGI2 applications in a server-agnostic way. (agronholm on IRC has pointed me to the "futures" module and PEP as a standardized API and will be experimenting for himself.)

"…your trying to push forward PEP 444 at this time is just making things uncertain again when they were starting to clarify that PEP 3333 would be the way forward at this point…"

PEP 3333 is a simple-to-implement organic evolution of PEP 333, and thus requires (or should require) little work on the part of the WSGI application developer or WSGI server author. PEP 444 is a clean break.

"Please just wait until the PEP 3333 issue has been settled, or do what you are doing well away from the WEB-SIG and not refer to it as in any way connected to WSGI."

I don't take being told not to do things well. ;)

Unfortunately, PEP 444 -is- WSGI, just not WSGI as you know it. The underlying API (environ dict, status/headers/body tuple return value) are the same, which should make familiarization with the differences less painful for the users experienced with WSGI 1.

"You may find the latter necessary anyway, as the WEB-SIG is likely to ignore you at this point and you aren't going to get any support from anyone."

I have found the opposite to be true; maintainers of the official PEPs on the Python site have contacted me, my taking over responsibility for PEP 444 was blessed by one of the original authors, and the amount of work I have accomplished creating a toolset to experiment with PEP 444 has been, even to me, stunning. The results have been stunning, too; my HTTP/1.1 server (now spec-complete) outperforms Tornado's HTTP server by 1 KRsecs on the same benchmark on the same machine. The implementation is clean, efficient, and simple enough to allow for rapid modification as PEP 444 evolves. (A total of ~170 opcodes in ~300 lines.) It's also Python 2.6+ and 3.1+ compatible and fully unit tested.

My fork of PEP 444:
https://github.com/GothAlice/wsgi2/blob/master/pep-0444.rst

marrow.server.http HTTP/1.1 server:
https://github.com/pulp/marrow.server.http

Graham Dumpleton said...

GothAlice

To say in respect of PEP 3333 that 'vagueness on certain critical issues for portability to Python 3' exist is wrong. Some people may have wanted it done differently, but as it stands it is still just as workable a solution for Python 3 as the original.

To say you have been granted control going forward is also wrong. As far as getting changes done to PEP 333 there was no one person in control and that is why no final decision really came about. This has not changed and sorry to say you are deluding yourself if you think you now hold all the cards.

I believe you will actually find that very few people have engaged with your discussion from the complete set of people whose input on this really matters. This is not because people trust you to come up with a better solution, but because people right now don't care and are sick of people pushing alternatives and the way that derailed the original process. I can't see therefore that you have any mandate and don't expect what you come up with to be blindly accepted. History of discussions on the WEB-SIG shows that will never happen.

Your solution also still doesn't address various shortcomings in the existing specification and so it doesn't matter if you wrap it in new clothing, it is still going to have the same issues.

Sorry if you think my language and dismissive is strong, but I have been working on this WSGI on Python 3 issue longer than anyone else and probably have a better understanding of it than anyone else.

GothAlice said...

Uh, wow.

"…but as it stands it is still just as workable a solution for Python 3 as the original"

I can only assume "it" refers to PEP 3333; what is the "original" solution to Py3K compatibility?

"To say you have been granted control going forward is also wrong."

I must have missed the part where I said I have control of PEP 333 or 3333. Or you missed that I only explicitly mention PEP 444; apologies for not making "PEP 444, for which I have been granted control…" more clear.

"I can't see therefore that you have any mandate and don't expect what you come up with to be blindly accepted."

I'm not an idiot; thank you for assuming otherwise. Also, see my above point about your misunderstanding of my previous post.

"Your solution also still doesn't address various shortcomings in the existing specification and so it doesn't matter if you wrap it in new clothing, it is still going to have the same issues."

I'm in the process of reading your previous posts tagged WSGI, and I'll be making notes as I go. I'll ensure the issues raised in your posts and elsewhere (wsgi.org, for example) are addressed.

I invite you to join #webcore on irc.freenode.net to discuss PEP 3333/444 with myself and agronholm.

"As far as getting changes done to PEP 333 there was no one person in control and that is why no final decision really came about. …but I have been working on this WSGI on Python 3 issue longer than anyone else and probably have a better understanding of it than anyone else."

These separate paragraphs belong together; when I see a problem I want to solve, I go out and solve it, even if it requires brute-forcing a problem domain. I'm helping to develop PEP 444 into a viable replacement for PEP 333 and 3333, and will be truly complete (and hopefully ratified) long after (again, hopefully) PEP 3333 has improved the WSGI1 ecosystem.

The lack of forward-progress on Python 3 compatibility across the Python ecosystem is frustrating, and I'll freely admit to not having the smoothest time in my marrow.* efforts (both from the code and from the community). The Python world seems to have a significant issue with those that attempt to "re-invent the wheel", even if the result can be an improvement. Core support for async in WSGI2 will be one of those improvements.

I agree with the first comment: release 3333 now using the (1, 1) version tuple. WSGI (2, 0) should begin to become ready for ratification around PyCon.

Also, I would love to see example code that conforms to PEP 3333—other than the CGI example in within the PEP itself—to examine API implementation details within a more complete environment.

Graham Dumpleton said...

GothAlice

You said 'PEP 444 has been re-named WSGI 2 (from "web3")' and you also said 'PEP 444, for which I have been granted control'. One can read this as meaning you believe you are setting the course of WSGI 2 which I would disagree with.

FWIW, there are already a number of people out there already claiming to be implementing WSGI 2, with different ideas of what that is. You have no more mandate than them for setting what WSGI 2 is and so why making an assumption that you can label it WSGI 2 is silly. All it is going to do is just confuse matters even more because of the growing number of competing WSGI 2 implementations. It is for that reason that I will object strongly and get quite cranky about anyone saying they are implementing WSGI 2. There are already too many things claiming to be that.

As to what I believe are shortcomings in the existing WSGI 1 specification and suggestions on how to resolve it, not all are really documented anywhere. Don't just assume that what you find in what I have written is everything.

As to PEP 3333 compliant implementations, the Apache/mod_wsgi support for Python 3.X has been available for something like 2 years now and is compliant with what the PEP 3333 specification documents. The only minor not is what wsgi.version is set to. The CherryPy WSGI server also has a Python 3.X implementation which should also be compliant with PEP 3333, again with perhaps wsgi.version not being 1.0 as PEP requires. That implementation has been available for perhaps year as well. Finally, I believe uWSGI has recently started claiming PEP 3333 compliance.

GothAlice said...

"FWIW, there are already a number of people out there already claiming to be implementing WSGI 2, with different ideas of what that is."

Any actual hard references? Again, I would like to see alternate implementations and so far they have only been alluded to.

"…one can read this as meaning you believe you are setting the course of WSGI 2 which I would disagree with. … You have no more mandate than them for setting what WSGI 2 is and so why making an assumption that you can label it WSGI 2 is silly."

With PEP 444 being already present on the Python site, it seems to me to already be further along than most.

"All it is going to do is just confuse matters even more because of the growing number of competing WSGI 2 implementations. It is for that reason that I will object strongly and get quite cranky about anyone saying they are implementing WSGI 2. There are already too many things claiming to be that."

Sadly, you being cranky isn't my problem. It is unfortunate, however, as you could be channeling your energy more productively.

"As to what I believe are shortcomings in the existing WSGI 1 specification and suggestions on how to resolve it, not all are really documented anywhere. Don't just assume that what you find in what I have written is everything."

Then to put it bluntly: then stop complaining and write it down. Or join IRC to discuss; my chat history logger works well.

"As to PEP 3333 compliant implementations, the Apache/mod_wsgi support for Python 3.X has been available for something like 2 years now and is compliant with what the PEP 3333 specification documents…"

I am aware of this. I'm also aware that you threw up your hands and gave up (disabling Py3K support in the shipping product). One of the commenters on that article wondered why you didn't release it anyway. As I have stated before, PEP 3333 can do a lot of good while maintaining gross compatibility between Py2K and 3K with the same WSGI1 API across both. Also as stated before, PEP 3333 should be ratified As Soon As Possible™. The support is there from the developers implementing it, the implementation is a refinement of PEP 333 w/ clarifications, and all can be happy.

PEP 444 / WSGI 2 is completely different in goals and approach.

Interestingly, I'll be exploring your "Definition #4" in a branch of marrow.server.http to see if the native strings idea is workable and can be made sufficiently efficient on the server-side. I like it, and if it can be efficiently implemented, I'll see about integrating it into PEP 444. It should simplify cross-compatibility more than the current "environ keys are native, values are bytestrings" definition.

Graham Dumpleton said...

@GothAlice

Apache/mod_wsgi does still include Python 3.X support in the code. It has not been disabled, it just isn't highlighted because of lack of ratification on whether PEP 3333 or anything else was going to be the solution. So, it is still there for people to use if they don't care about the uncertainties over what WSGI for Python 3.X will be.

See 'http://code.google.com/p/modwsgi/wiki/SupportForPython3X'.