In mid-September, I requested a grant from the Python Software Foundation to port WebOb to Python 3. The proposal was as follows:
Porting WebOb To Python 3 ========================= Proposal submitted to the Python Software Foundation September 16, 2011 Abstract -------- WebOb ( http://docs.webob.org/en/latest/index.html ) is a library used to make creating Python web applications easier. It contains request and response abstractions which automate tasks such as setting response headers, setting cookies, decoding headers and cookies, generating ETags, and working with query strings. It is based on the WSGI protocol. WebOb was created by Ian Bicking, and is now maintained primarily by Sergey Schetinin. WebOb is currently used in the following major systems: - Pylons (https://docs.pylonsproject.org/projects/pylons_framework/dev/) - Pyramid (https://docs.pylonsproject.org/docs/pyramid.html) - Google App Engine (http://code.google.com/appengine/) - Bobo (http://bobo.digicool.com/) - Restish (http://ish.io/projects/show/restish) - TurboGears 2 (http://turbogears.org/2.0/docs/) It is also used by other less-well-known frameworks and in many WSGI middleware implementations. Currently, WebOb is limited to running on the following platforms: CPython 2.5, CPython 2.6, CPython 2.7, PyPy 1.5 and 1.6, and Jython 2.5.2. This proposal requests funding to port WebOb to Python 3. I propose to do this work. After the work is completed, a version of WebOb will be available which is compatible with both Python 2 and Python 3. Why Funding Is Needed --------------------- The funding will service two high-level requirements: - If Python 3 is to be maximally useful to web developers, they need tools that make working with WSGI easier. Working with "bare" WSGI as an application development platform is error-prone and difficult. The lack of a shared request/response abstraction on Python 3 leads to duplication of effort and dissatisfaction with Python as a web application development platform. Existing Python 3-compatible web frameworks (Bottle, CherryPy) each implement their own request/response abstractions which are not divisible (or at least not separated) from their respective framework. Continuation of this pattern will lead to further fragmentation of the Python web development community as time goes on. - If Python 3 is to be relevant to existing Python WebOb-based web framework authors, a Python 3-compatible WebOb must be made available to them. Existing web frameworks which depend on WebOb cannot be ported to Python 3 without being able to depend upon a Python 3-compatible WebOb implementation. WebOb currently does not run on Python 3. Creating a version that will run on Python 3 will satisfy the above requirements. The person or group who successfully ports WebOb to Python 3 must have the following skills: a deep understanding of HTTP and related specifications, a deep understanding of WSGI, mastery of Python 2, mastery of Python 3, mastery of testing tools, and a deep understanding of WebOb design goals. There aren't very many people with the prior qualifications who aren't already engaged writing Python web applications full-time on Python 2. They can't use Python 3 without having a suitable toolchain available, and therefore they don't see much benefit in using it. Making WebOb available on Python 3 is also a rather boring and unrewarding task. As a result, it has been difficult to convince anyone to to take the time from their schedule to provide an initial Python 3 port. It's a bit of a chicken-or-egg scenario. Funding a port of WebOb to Python 3 will break the chicken-or-egg cycle, and make Python 3 a viable platform to the people who use WebOb, and, eventually, people who use frameworks based on WebOb. I submit this proposal mainly on the advice (via Twitter) of Jesse Noller, who replied to my public reports of trying to port WebOb on my own time with a suggestion that I apply for a grant (http://twitter.com/#!/chrismcdonough/status/108292018182172673). Strategy -------- The strategy will be to have a single version of WebOb that will run on both Python 2 and Python 3 until such time as Python 2 compatibility can be retired. A Python 2-compatible version of WebOb must continue to exist for the foreseeable future, as a large number of existing projects depend upon the combination of Python 2 and WebOb. Both bugfix and feature releases of WebOb will need to be made indefinitely for the Python 2 platform. A fork of WebOb to make it run on Python 3 that abandons Python 2 is untenable. There is no one willing maintain a Python 3-only version of WebOb. Likewise, there is no one willing to synchronize changes made to separate Python 2 and Python 3 versions of WebOb, and having the two versions fall out of sync is undesirable for forward and backward compatibility purposes. Therefore, we propose to use a single codebase to support both Python 2 and Python 3. This codebase will work under CPython 2.5, 2.6, 2.7, and 3.2, PyPy 1.5 and 1.6, and Jython 2.5.2. Since WebOb is very heavily based around I/O (taking data from the web and presenting it to the developer in a form he can work with), automated porting strategies such as 2to3 don't help very much; lots of code that does encoding and decoding between bytes and text needs to be changed "by hand". Likewise, details of the new WSGI specification detailed in PEP 3333 require attention to detail that fall outside the scope of any automated translation. 2to3 automated translation is also costly in terms of library installation wall-time. Therefore, the WebOb codebase will "straddle" Python 2 and Python 3 using a shared subset of both languages and will not rely on any automated translation tools. WebOb has no non-stdlib dependencies that aren't used solely for testing purposes. We will rewrite any existing tests that have Python-2-only testing requirements to use Python 3-compatible analogues. Prior Work ---------- In March of 2011, the Pylons Project sprinted at PyCon with the goal of providing 100% unit test statement coverage to WebOb. This goal was reached and laid the groundwork for being able to port to Python 3 by making it possible to have some level of confidence that Python 3-compatibility-related changes do not break existing code. In the summer of 2011, Google sponsored student Joe Dallago to help port WebOb to Python 3 via their Google Summer of Code program. This project was mentored by people from the Pylons Project. Joe did the best he could, but the task was too complex to be completed successfully within the timeframe of GSOC. However, his work at https://bitbucket.org/jayd3e/webob-1.1-py3 will serve as a basis (or at least as a guideline and reference) to a funded porting effort. Schedule and Budget ------------------- The amount of work required to perform the port as described above in "Strategy" is estimated to be 80 hours. The work can be broken down like this: - Remove all existing deprecated methods which promise results as binary data (``str_GET``, ``str_POST``, ``str_params``). - Redesign existing "MultiDict" strategy to not encode/decode at lookup time, but instead to decode to text at request.POST / request.GET / request.params dictionary creation time. - Redesign existing cookie encoding/decoding strategy to work in terms of text instead of binary. - Redesign existing request and response descriptor implementations which take environ data and return binary data to expect and return text instead. - Get all existing WebOb tests passing on Python 3. - Functionally test on Python 2 and 3 under wsgiref, the CherryPy WSGI server, and mod_wsgi. On Python 2, functionally test under paste.httpserver. - Run "in the wild" middleware, frameworks, and applications under the new codebase on Python 2, to ensure we haven't broken anything (use of non-API-but-probably-should-be-API methods/classes/functions). - Ensure 100% test statement coverage on both Python 2 and Python 3. - Update documentation to detail differences between usage of WebOb on Python 2 and Python 3; also note any behavior differences from prior releases. - Work with Sergey Schetenin to merge the code into the WebOb mainline repository. We have already discussed the need to support Python 3, so this will be a code review process, not a process of defending the need for a Python 3-compatible version. Coding work will be done on a public Bitbucket fork of the main WebOb repository. The work will begin on October 1 and will finish on October 31. I am requesting a total of of $3000.00 for this effort, to be paid on or before November 30. Risks ----- There is not much "big design" thought required to do this work; just lots of "little design" requirements. Most of the truly hard legwork has already been done by the Python core team (wsgiref), by Phillip Eby in PEP 3333, by Joe Dallago in his GSOC efforts, and by Graham Dumpleton and Bob Brewer in mod_wsgi and the CherryPy WSGI servers, respectively. There is little risk of getting the design wrong, as the WebOb design already anticipates most requirements of encoding and decoding web data from binary to text. The only real risk is to existing users of WebOb under Python 2. We need to make sure it continues to work for them and isn't backwards incompatible for indefensible reasons. I'll attempt to make sure of this by running "in the wild" middleware under the new implementation, and fixing things as appropriate. Qualifications -------------- I am the primary author of the Pyramid web framework, which uses WebOb as a base. I have contributed regularly to WebOb, and I have collaborated with its maintainers many times. I have served as a Google Summer of Code mentor this year to a student working on WebOb. I am the original author of Python PEP 444 ("Web3", a WSGI replacement protocol), although I am no longer its maintainer. I have been developing Python web applications for 11 years. Conclusion ---------- For a relatively small amount of funding, I believe we can make Python 3 a more pleasant web development platform, recognizable to people who have a history developing web applications under Python 2. New users will also benefit by having a reasonable request/response abstraction rather than dealing with "bare" WSGI or being tied to a particular framework implementation when they use Python 3.
I'm happy to report that this proposal was accepted by the PSF, and that we've now released a Python 3 compatible version of WebOb to PyPI .
Here's a report-out on the concrete aspects of the proposal:
DONE: the codebase works under wsgiref and mod_wsgi; the CherryPy WSGI server is not yet PEP 3333 compliant so it could not be tested.
DONE: The Pyramid, Pylons, Restish, and ToscaWidgets 2 test suites were ensured to run using the new code (not all tests passed but stakeholders with failing tests were advised of potential backwards incompatibilies).
DONE: We have 100% test coverage under Python2, but not under Python 3. Some of the code executes solely on Python 2 and some solely on Python 3 and no existing coverage tool allows us to conditionalize coverage like this.
DONE: WebOb docs adjusted thanks to Sergey.
DONE WebOb moved to the Pylons Project on GitHub, and release made.
The WebOb changelog has the gory details about changes and backwards incompatibilities. You can see the (monster) diff at https://github.com/Pylons/webob/compare/1.1-branch...master
Thanks much to the PSF board, Sergey Schetenin, and Joe Gallago.