The PSF work in the Google Highly Open Participation project seems to be going well! I personally have seen two tests get rewritten from old-style stdout comparison to doctest with another three being actively worked on.
But if there is one thing this experience has taught me is that a doc on how to run the tests and write tests for Python is badly needed. There has been a consistent misunderstanding over why the tasks to ditch the Lib/test/output tests for doctest are desired. There has also been questions about how to best go about using doctest and such. And since so many of the entry-level things people can work on are test-related it would be really helpful to have a doc that people can read to understand how everything is structured.
All of this has reminded me about my mid-term plans for my Python work. I am not sure if I have stated this publicly even though I have been thinking it for a while, but I am planning to really focus my dev time in the future to helping to make Python easier to hack on and maintain. Once my current "cutting edge" Python commitments are complete (importlib and the Py3K stdlib reorg) I am shifting my focus to stuff that is not as time critical and more specifically focused on helping to make developing Python easier.
I think first I will write up some docs. Beyond just the testing doc I want some basic stuff on how to report and review a bug report, how to create and review a patch, and how to get commit privs and do an actual commit. There should probably also be a doc on how to get your environment set up and an intro to the directory structure. I am hoping to keep these docs at a high enough level that I don't get bogged down in detail but have enough info so that if we have people show up at a sprint or a bug day they can just read those docs and know what is needed when one takes on a bug report or patch.
With hopefully more people helping it will probably be close to time to re-evaluate the workflow we have for issues. The current one was inherited from SourceForge and it just doesn't work well for our needs. Now that we control our own issue tracker I want to take the opportunity to make it easier to manage issues and help make it easy to stay on top of things so we don't end up with a backlog of open issues.
Next I want to clean up the testing framework. As it stands it really sucks having all of the tests in a single directory. They should at least be divided into language, built-ins, and stdlib. Might also toss in a prerequisite package for those tests that have to pass in order for the test suite to even be considered in a sane state are kept in a single location.
I then want to do something that the PyPy folk wanted ages go. Tests should specify if they are blackbox or whitebox tests. It sucks that we have tests that are specific to CPython when various implementations of Python use the unit tests as a way to verify their implementation. It should be easy for them to know what tests they don't need to implement. If, when the time comes, the PyPy, Jython, and/or IronPython communities are willing to help me mark up the test suite then I am willing to write the code to deal with this.
To go along with this, it should be easy to specify tests that are platform-dependent within the test itself. This goes for the module level as well as at least the class level, if not the method level.
Each test should also have a way to specify what module they are testing (so that failed imports of support code shows up as a test failure instead of a skipped test). This will also help to make sure all modules have a test in the first place.
And finally (in terms of changing code in tests), every test should have a standardized function to call. Not every function has a test_main() function, but it should. Might also be good to have it take an optional verbosity argument or something instead of reading off an attribute on test.test_support.
I want regrtest.py to be greatly simplified from what it is now. It really should only facilitate in executing tests, not help judge if a test passes (e.g., output comparison tests) or whether a test should be run (e.g., whether a specific test is platform-specific or not). Only stuff that is not unique to any one test should be in regrtest.py. If the above implemented that should be doable.
With the tests all organized we will need to know how effective the tests are. Getting code coverage reported daily would be nice. I know Walter Dörwald has his coverage report, but I want something on trunk that is run nightly and is more visible. I want it so that if someone feels like contributing to Python but don't have a personal itch to scratch they can either look for a bug or patch to review or work to improve the code coverage on some module.
With all of this in place it is probably time to look at the build system. I hate autoconf and don't love GNU Make. I would love to replace them with another build system that is a lot easier to maintain.
With the maintenance of what we have taken care of, it should then be time to simplifying what we do have to maintain. The thing that specifically comes to mind on this topic is the conversion from CST to AST by the compiler. Right now it is handled by hand-coded C. What I would rather see is a Python-implemented domain-specific library (not language!) handle the mapping of grammar to AST that generates C code. Basically I want to minimize the amount C code in Python to only stuff that can't be auto-generated by Python code.
Obviously this is a lot of stuff to do and it will take years to complete. But all of this stuff has been on my todo list for ages so I suspect I will actually do a lot of them. I really want to make contributing to Python as easy as possible. Not just so that I have to do less grunt work and thus get to do more nutty things (e.g., trying to develop new bytecode), but also so other people can feel the same satisfaction I do from helping with the development of Python.
