2010-01-16

The importers project is now public

For my PyCon talk on importers, I needed a running example. After asking online what people wanted as an example I ended up creating an importer that uses sqlite3 databases ... which I then scrapped and rewrote from scratch after realizing that there was a lot of boilerplate that I could abstract out. I decided to some ABCs to make it easy to write other importers (specifically a zipfile one), much like the ones I have in importlib. This not only helped to simplify the code, but it let me easily conceptualize what is and is not consistent between importers. Knowing what is common helps me know what I need to cover at PyCon.

But there is no need to keep this code to myself, and so the importers project has been created. The project is hosted at Google Code, uploaded to PyPI, with docs at packages.python.org. The code has a bunch of ABCs that let you create finders and loaders as long as you can specify a few simple operations from the perspective of a file path (e.g. a file exists, reading from a path, etc.). This makes it very easy to create new importers that use different archive formats as you simply need to manipulate paths and read from the archive; the ABCs do all of the fancy import work for you. I also included both the sqlite3 and zip importers in the package in case anyone wanted to use them. I have also gone ahead and include a lazy loader mixin that works with other loaders. This is the same lazy loader I blogged about in the past, but with proper unit tests.

Thanks to the importlib dependency, this is Python 3.1 only. My hope is that feedback on the code in the project will be such that I feel confident in moving the code into importlib. While I was able to get importlib into the standard library w/o following the typical one year waiting period, I am not skirting that unofficial rule anymore. You can consider the importers project purgatory for future importlib code (assuming feedback is positive).

This has also been the first project I used Distribute with. It all went rather smoothly, even running under Python 3. What was really handy was the upload_docs command once I used the --upload-dir flag (for some reason my setup.cfg was not doing what I needed). In other words, good work Tarek and the people helping him with Distribute!