Classifying unit tests

A while back the PyPy folk said they would love to have support for classifying what tests in the stdlib were implementation-specific. That would allow them to run the test suite without having to keep their own list of tests that they can never pass. Don't know if they are still interested in this, though.

Having written some new context managers for test.test_support and read about Collin Winter's rework of unittest, I am inspired enough to at least blog on the topic.

What I see is test.test_support.classify being a decorator. Based on keyword arguments one could specify what operating systems the test is expected to work on specifically along with whether it is implementation-specific:

@classify(os=["win", "darwin"], implementation=True)
def test_foo(): pass

That would probably set a __testing__ attribute on the method with a dict containing specific keys and values or a set with very specific values. Accessor methods could be provided, but considering only test runners should care about this info it probably isn't really needed.

For doing an entire class there are two options. One is to use a metaclass whose constructor takes the same kind of arguments and then applies them to every method. The other is to provide a function that returns what should be set for __testing__ on the class. Either one would work although I think unittest.TestCase is still a classic class.

The only other thing I have been planning on doing for unit tests is changing how ImportError is handled by regrtest. As of right now if any form of import failure occurs then a test is skipped saying a certain module was not found. But what should really happen is that the test is skipped if the module to be tested is not found. Any other import failure should be an error.

To handle that one needs a function that takes in a module name to import and returns that module. Obviously some support would be need for not only for ``import ...`` imports but ``from ... import ...``. But it is very straight-forward to write a function to do that; trick is the interface.