PEP 3147 (PYC Repository Directories) are causing me grief. When I created the ABCs for importlib I did not expect import to change significantly. But here I am, in a situation where the handling of bytecode files has shifted significantly, making the abstractions I created for importlib not work quite as well as they did when Python 3.1 came out. That means something needs to change, but that's not easy when it comes to Python's standard library.
It is said that the standard library is where code goes to die, developmentally speaking. Once your code lands in the stdlib, it's hard to change. You end up not knowing who is using your code or how. But being in the stdlib guarantees that at least one person is somewhere in the world. And if you change something without providing an upgrade path, you are going to have some pissed users. Now, you could argue that as long as you work for three releases (e.g. 2.5 through 2.7), you're safe as that is roughly 4.5 years of compatibility (ask the CentOS/RHEL people and they would say four versions thanks to their 2.4 dependency, but that is a bit nuts as we are not even doing security releases at this point for 2.4). But that still requires that you provide some way to be compatible through three versions of Python. How the heck should I make it so that people using the ABCs from Python 3.1 still work unmodified -- albeit with warnings being triggered -- in Python 3.4?
First, let's look at what caused the problem. When I created the ABCs I assumed that there were primarily two situations: you only cared about source or you cared about source and bytecode. The thinking was that if you wanted to do some wonky source transformation or you were working with a VM that didn't use CPython bytecode files (i.e. all other VMs), then you would want to use the PyLoader that only used source. Otherwise you used the PyPycLoader for source & bytecode usage.
PEP 3147 makes the split more along the lines of source or source-less importing. The PEP leads to Python emphasizing more than it did that bytecode files are really an optimization that happen to have been used by some as a distribution mechanism (there was actually a bit of back-and-forth about removing bytecode-only/source-less import support, but enough people used this to continue supporting it unfortunately). That means that my division in the ABCs was no longer in alignment with how things will be viewed by importer authors going forward.
The improper division is made even more acute by the fact that the PyPycLoader exposes bytecode details that do not extrapolate into a new PEP 3147 world. For instance, PyPycLoader.bytecode_path() in Python 3.1 returns the path to the bytecode file next to the source file. But with PEP 3147 you have the cached file path when there is source code along with the path to the bytecode file in a source-less import. The bytecode_path() method now must branch its logic based on whether there is any source code. That's really unfortunate for anyone who implemented their own bytecode_path() as it will no longer operate properly in the face of PEP 3147 as how everyone else expects it to work. So in this case I exposed too much to the user of the ABC.
I also messed up by introducing new methods that took module names instead of "file" paths. Everyone assumes that you are working on a file system or something that resembles one, so adding the overhead of having all of the methods have to resolve a module name to a path name to perform their operations -- e.g. PyPycLoader.source_mtime() 99% of the time will need to stat a file path, not some abstracted thing where you can index by module name alone. That's annoying as it leads to a lot of boilerplate in all of the methods I introduced beyond PEP 302.
Not using file paths is also unfortunate when it comes to using a loader as a file system abstraction. Theoretically, if you have data files with your package code, you should be able to use __loader__.get_data() to read it and not have to worry about any other details such as if you are reading from a zip file or an actual file system. Had I used file paths for things such as source_mtime() then people would have gained modification time info on files they stored with their package code.
So how do I fix my initial design mistakes while also fixing issue that PEP 3147 have brought up, all while letting importers written for Python 3.1 to continue to function? First, I am going to deprecate PyPycLoader and introduce PycLoader. Doing this will let me keep PyLoader to handle source imports, but make using cached bytecode files an optimization detail that I do not have to directly expose to the loader author. And adding PycLoader allows me to still support source-less loaders without cluttering the code for PyLoader (I might actually not even add PycLoader as I really don't like source-less imports and do not want to promote them by making it easier to use them beyond what is necessary). I could simply deprecate both PyLoader and PyPycLoader and introduce new ABCs named SourceLoader and BytecodeLoader, but that leaves people who have already written importers in a bad position of having to maintain two class definitions which would be really unfortunate.
For PyLoader, get_filename() will fully supplant source_path(). This was already the plan as it makes a loader more compatible with runpy, but this just makes it that much easier to have the support inherent in the design of the ABC. That's very minor as get_filename() can be written to not be abstract and to call source_path() as it does now. But what I can do is make that usage trigger a deprecation warning (all warnings I mention here will start off as a PendingDeprecationWarning and then shift to a DeprecationWarning, and then finally be removed). For people who want to easily support both Python 3.1 usage where source_path() was the thing to use and Python 3.2 and newer, all they need to do is define get_filename() and alias source_path() to it in the class definition. That will override the warning-raising version of get_filename() in Python 3.2 and newer but still provide source_path() for Python 3.1.
To support cached bytecode files as introduced in PEP 3147, an optional method will be introduced: path_mtime(). This will take a path and return the modification time of the path, raising IOError if the path does not exist (just like ResourceLoader.get_data()). In order to support Python 3.1 loaders, if path_mtime() is not defined then source_mtime() will be used and a warning will be raised. To support both 3.1 and 3.2 loaders, you can simply write your own source_mtime() that calls get_filename() to get the path to the source code and then call path_mtime(), returning None instead of letting the IOError propagate (returning None for source_path() and bytecode_path() was a bad move on my part; I prefer EAFP over LBYL).
A similar approach to path_mtime()/source_mtime() would be used for adding set_data() to replace write_bytecode(). The reason for naming the new method set_data() is for naming consistency with get_data(), even though I would rather name it write_data(). And once again, moving to file paths over module names adds to the file system abstraction.
In all of this, bytecode_path() goes away. Why? Because it's not needed thanks to imp.source_to_cache(). As I said earlier, bytecode files are viewed more as optimizations than ever before, meaning that Python should control that optimization and not the user for the sake of consistency and any future changes to that optimization. It also allows other VMs to signal that bytecode is not supported by simply having imp.source_to_cache() return None, making PyLoader work on any Python VM properly out-of-the-box.
This does means that with PyPycLoader being replaced by PyLoader with some optional methods leads to Python 3.1 PyLoader implementations not having bytecode support. It's unfortunate, but should be a minor loss. But if one really needs that bytecode support, you can conditionally choose which ABC to use as a base class and make sure you implement all of the needed methods to work in Python 3.1.
Overall I think it's a reasonable transition plan. Backwards-compatibility is kept from Python 3.1 until I choose to rip out support for the old way without having to do major contorting. Plus it helps future-proof the ABCs such that they won't have to go through this transition period again.
Showing posts with label importlib. Show all posts
Showing posts with label importlib. Show all posts
2009-06-02
Lazy imports in 25 lines of code
For some unknown reason to me, I found myself thinking about lazy importing on my way home from school. Now I have never actually dived into the topic before so I had no preconceived notions of how to solve the problem. I do know that people like to have imports be lazy so that the startup time of their applications are not dominated by imports of modules that they do not need immediately, if at all. Many projects have replicated their own solution to this problem such as bzr and hg. This has even gone as far as Greg Ewing proposing actual syntax support on python-ideas and Christian Heimes using lazy imports as a motivating factor for post import hooks.
While the solution varies, the general agreed upon solution is to call some custom function that will return a module proxy. That proxy triggers the actual import once the module is actually used in some meaningful way, such as when an attempt is made to access an attribute. I also assumed that people either used a custom importer (meta path or not) or a custom
Going in blind on this, I started to think about what it would take to pull this off using importlib in Python 3.1 for Python source. I was not going to worry about built-in modules and the like as importing those types of modules are much cheaper compared to importing Python source or bytecode. And I wanted to solution to be as simple as possible since I was doing this for fun, not as an actual robust solution to a problem I had.
With those goals in mind, I realized that I needed to come up with a lazy loader that could be subclassed for ease of use. I also knew I needed to come up with module proxy object that would trigger an import the instant an attempt was made to access an attribute. Seems simple enough, but obviously it isn't obvious.
First problem I had to solve was how the module proxy would get at the real loader when it was eventually needed. That was quickly solved by assigning the lazy loader to
With that problem solved I turned my attention to the module proxy. After some mishaps with trying to dynamically set
The final problem to work out was how to make sure that the module instance was what got initialized by the import so that all pre-existing references to the module instance worked after the import. The obvious way is to make the module proxy act as an actual proxy; you could store the initialized module in the proxy and forward all attribute requests to the initialized module using
That more elegant solution is module reloading. If you look at PEP 302 you will notice that to support module reloading that a loader is expected to reuse the module found in
It turns out that all of this was enough for a proof-of-concept to work! By creating a class that inherited both the lazy loader and importlib's private Python source file loader I was able to defer actual importing of a source file until an attribute on the module was accessed. And the best part was that the only requirement of the real loader was that it follow the rules set out in PEP 302 which importlib provides with
To see the proof-of-concept which is NOT PRODUCTION QUALITY -- remember it is using private APIs from importlib which could easily disappear at any time -- see this paste bin. I am sure the code should be more robust somehow, but I am rather pleased with the solution is only 20 source lines and appears to work at least in a really dead-simple example. If people actually like this approach and think they would find it useful then leave a comment and I will see if I can be persuaded to package it up and write the proper tests so that I would be willing to put this up on the Cheeseshop.
While the solution varies, the general agreed upon solution is to call some custom function that will return a module proxy. That proxy triggers the actual import once the module is actually used in some meaningful way, such as when an attempt is made to access an attribute. I also assumed that people either used a custom importer (meta path or not) or a custom
__import__ function. But beyond that I had no clue how people solved the issue. The importer solution can be seen in a cookbook recipe and the custom function can be seen in peak.util.import (source).Going in blind on this, I started to think about what it would take to pull this off using importlib in Python 3.1 for Python source. I was not going to worry about built-in modules and the like as importing those types of modules are much cheaper compared to importing Python source or bytecode. And I wanted to solution to be as simple as possible since I was doing this for fun, not as an actual robust solution to a problem I had.
With those goals in mind, I realized that I needed to come up with a lazy loader that could be subclassed for ease of use. I also knew I needed to come up with module proxy object that would trigger an import the instant an attempt was made to access an attribute. Seems simple enough, but obviously it isn't obvious.
First problem I had to solve was how the module proxy would get at the real loader when it was eventually needed. That was quickly solved by assigning the lazy loader to
__loader__ and treating the class as a mix-in. Assuming the lazy loader class name was LazyMixin and I set up the inheritance correctly, I could get at the real loader from within the module proxy with super(LazyMixin, self.__loader__). By re-assigning the resulting object to __loader__ I would continue to have the actual loader doing the loading on the module as well have a consistent way to get at the loader I was after. Assuming a module proxy class named LazyModule, the code looks like this:
class LazyMixin:
def load_module(self, name):
module = LazyModule(name)
module.__loader__ = self
sys.modules[name] = module
return module
With that problem solved I turned my attention to the module proxy. After some mishaps with trying to dynamically set
__getattr__ I had an epiphany: I could simply reset __class__ to a class that lacked the __getattribute__. By having both the lazy module class with __getattribute__ defined and the module class that __class__ gets set to inherit from types.ModuleType I could make sure that the module instance was always consistent in terms of its type. This all led to the following module objects:
class Module(types.ModuleType):
pass
class LazyModule(types.ModuleType):
def __getattribute__(self, attr):
self.__class__ = Module
self.__loader__ = super(LazyMixin, self.__loader__)
self.__loader__.load_module(self.__name__)
return getattr(self, attr)
The final problem to work out was how to make sure that the module instance was what got initialized by the import so that all pre-existing references to the module instance worked after the import. The obvious way is to make the module proxy act as an actual proxy; you could store the initialized module in the proxy and forward all attribute requests to the initialized module using
__getattribute__. But that has a performance penalty that seems unnecessary and I had a gut feeling there had to be a more elegant solution.That more elegant solution is module reloading. If you look at PEP 302 you will notice that to support module reloading that a loader is expected to reuse the module found in
sys.modules when performing a load. This makes sure that existing references to a module continue to work as the module instance doesn't change, just its __dict__. So by sticking the module proxy instance into sys.modules the real loader would use that module object for the initialization, letting all pre-existing references to continue to be valid.It turns out that all of this was enough for a proof-of-concept to work! By creating a class that inherited both the lazy loader and importlib's private Python source file loader I was able to defer actual importing of a source file until an attribute on the module was accessed. And the best part was that the only requirement of the real loader was that it follow the rules set out in PEP 302 which importlib provides with
importlib.util.module_for_loader. And an even cooler thing is that since the lazy loader is a mix-in that only overrides load_module all other methods from the real loader are accessible, meaning the lazy loader allows for the use of introspection APIs such as importlib.abc.InspectLoader to continue to work! And the greatest perk of all is that since it works with loaders it is completely transparent to the import statement, removing any need to use a custom function to make this work!To see the proof-of-concept which is NOT PRODUCTION QUALITY -- remember it is using private APIs from importlib which could easily disappear at any time -- see this paste bin. I am sure the code should be more robust somehow, but I am rather pleased with the solution is only 20 source lines and appears to work at least in a really dead-simple example. If people actually like this approach and think they would find it useful then leave a comment and I will see if I can be persuaded to package it up and write the proper tests so that I would be willing to put this up on the Cheeseshop.
2009-03-08
Importlib is now useful to other people
I have always had two goals for importlib. The docs for importlib say they were to provide a reference implementation of import and to make it easier for people to implement their own importers. The former goal is sort of true; the real goal is to not just be a reference implementation of import but to actually become THE implementation of import. The latter goal, though, is spot-on and I finally checked in a big chunk of code tonight that gets me closer to helping other people really harness the flexibility of import.
The cron job to rebuild the docs has not kicked in yet (I think it runs twice a day), but once it does you will discover there is now a new module: importlib.abc. Within the module there are ABCs for everything specified in PEP 302. Back in January I asked for help to name those classes. The fruition of that discussion is now finally live.
But simple ABCs to require load_module exist as a method isn't THAT helpful. What I really wanted to provide was something to make it as easy as possible to write their own custom loaders such that they didn't have to worry about the little details that are consistent between all loaders (and there a lot of details; just look at the "see also" section of the importlib docs). Way back in August 2007 I came up with an idea called handlers which would deal with stuff like setting __file__, making sure bytecode is recreated when it's older than the source code, etc.
Unfortunately handlers turned out to be somewhat burdensome. They required a bunch of information upfront to be passed into them that loaders had to provide. On their own they couldn't do much and just became internal delegates for loaders. So I set about trying to come up with a way to merge the handler concept into loaders.
And it turns out PEP 302 got me part way there. If you look at what it takes to import Python source code (I am ignoring bytecode), you essentially create an empty module, read in the source code, compile the source into a code object, and then execute the code object in the __dict__ of the empty module. After that it's just details like __file__ and such. When you take these basic steps you will notice they roughly align with some APIs that PEP 302 defined as optional protocols loaders could implement. So I asked myself if I could somehow harness the optional protocols to get all the information I need so that I can simply provide a loader that uses those protocols.
Looking at source only, it would seem like the answer is "yes" since there is a get_source method which obviously returns source code for a module. One would think then an implementation for load_module would simply call get_source, compile it, and then use it to create a module. But of course life is not simple.
First of all, as I have discussed before, reading source from disk was not working for me as I didn't have a simple way to get source from disk in the proper decoding thanks to PEP 263. Everything I came up with was on the complicated side. That meant get_source was not exactly a nice thing to rely upon.
But the other issue of relying on get_source is it doesn't tell me the path the source came from. That's needed for __file__ to be set. That completely kills solely relying on get_source.
You could potentially rely on another part of PEP 302 which defines a get_code method which is supposed to return the code object for a module. But that puts more burden on a developer than I wanted to.
At this point I realized I was probably going to have to add to the APIs that PEP 302 provided. I didn't want to do this as that just makes it that much more difficult to implement a loader, but I realized that the PEP 302 protocols simply did not provide all the information needed to create a module from scratch. So I started to think about what the minimum amount I needed to add to the API.
And I thought. And I thought. And I thought. Whenever I have blogged about APIs while working on importlib, it has been in regards to this conundrum of building off of PEP 302's protocols with something that is simple and useful no matter what the storage back-end for modules happened to be.
Eventually I had an epiphany. Using get_source was not an option because it was missing the path to set __file__ to. Somehow the loader needed to have some concept of paths to set __file__ in some meaningful fashion, even if it wasn't really a file path. If the loader provided some concept of a path, then I could use the loader as if it was using a file-based back-end. If I went with that assumption I could use get_data from PEP 302 in order to get at the source code; get_data(source_path('module')).
But I hesistated for a long time at using get_data to fetch source code. Having get_source sitting right there was just so tempting! But then I started to consider how to handle reading bytecode. Should I have a get_bytecode method? But I then run into the problem of needing a bytecode_path method to be able to set __file__ probably for modules loaded from bytecode (and when no source is available; new semantics of 3.0). Going that route means I would have added source_path, bytecode_path, and get_bytecode just to read source and bytecode. This still doesn't deal with getting the modification time for source to see if the bytecode is stale or writing bytecode to the storage back-end through the loader.
Realizing that going the get_bytecode route duplicated functionality needlessly, I went with using get_data to read source and bytecode based on what source_path and bytecode_path return. This keeps the functionality per method simple and mostly unique. It's definitely a "misuse" of get_data as it was not meant to be used this way, but it makes sense and keeps the API simple.
With all of this put together I can provide an ABC that implements load_module in terms of the PEP 302 protocols and just a couple of other methods that handles all the stuff that is not specific to the back-end being used to store the source or bytecode. For instance, to implement a source loader, one needs to implement:
But the big one is when handling source and bytecode together. There you add the above methods plus:
With all of this done, that leaves just two parts left to implement for the public API (get_source in terms of get_data and source_path along with a decorator I found useful). Once that is done, importlib is semantically done for CPython. I do need to talk to Jython, IronPython, and PyPy to see what they might be missing that I rely on from CPython such that if someone implements a source/bytecode loader it will still work on those VMs even if bytetcode happens to be present (this worry is thanks to PEP 370).
The cron job to rebuild the docs has not kicked in yet (I think it runs twice a day), but once it does you will discover there is now a new module: importlib.abc. Within the module there are ABCs for everything specified in PEP 302. Back in January I asked for help to name those classes. The fruition of that discussion is now finally live.
But simple ABCs to require load_module exist as a method isn't THAT helpful. What I really wanted to provide was something to make it as easy as possible to write their own custom loaders such that they didn't have to worry about the little details that are consistent between all loaders (and there a lot of details; just look at the "see also" section of the importlib docs). Way back in August 2007 I came up with an idea called handlers which would deal with stuff like setting __file__, making sure bytecode is recreated when it's older than the source code, etc.
Unfortunately handlers turned out to be somewhat burdensome. They required a bunch of information upfront to be passed into them that loaders had to provide. On their own they couldn't do much and just became internal delegates for loaders. So I set about trying to come up with a way to merge the handler concept into loaders.
And it turns out PEP 302 got me part way there. If you look at what it takes to import Python source code (I am ignoring bytecode), you essentially create an empty module, read in the source code, compile the source into a code object, and then execute the code object in the __dict__ of the empty module. After that it's just details like __file__ and such. When you take these basic steps you will notice they roughly align with some APIs that PEP 302 defined as optional protocols loaders could implement. So I asked myself if I could somehow harness the optional protocols to get all the information I need so that I can simply provide a loader that uses those protocols.
Looking at source only, it would seem like the answer is "yes" since there is a get_source method which obviously returns source code for a module. One would think then an implementation for load_module would simply call get_source, compile it, and then use it to create a module. But of course life is not simple.
First of all, as I have discussed before, reading source from disk was not working for me as I didn't have a simple way to get source from disk in the proper decoding thanks to PEP 263. Everything I came up with was on the complicated side. That meant get_source was not exactly a nice thing to rely upon.
But the other issue of relying on get_source is it doesn't tell me the path the source came from. That's needed for __file__ to be set. That completely kills solely relying on get_source.
You could potentially rely on another part of PEP 302 which defines a get_code method which is supposed to return the code object for a module. But that puts more burden on a developer than I wanted to.
At this point I realized I was probably going to have to add to the APIs that PEP 302 provided. I didn't want to do this as that just makes it that much more difficult to implement a loader, but I realized that the PEP 302 protocols simply did not provide all the information needed to create a module from scratch. So I started to think about what the minimum amount I needed to add to the API.
And I thought. And I thought. And I thought. Whenever I have blogged about APIs while working on importlib, it has been in regards to this conundrum of building off of PEP 302's protocols with something that is simple and useful no matter what the storage back-end for modules happened to be.
Eventually I had an epiphany. Using get_source was not an option because it was missing the path to set __file__ to. Somehow the loader needed to have some concept of paths to set __file__ in some meaningful fashion, even if it wasn't really a file path. If the loader provided some concept of a path, then I could use the loader as if it was using a file-based back-end. If I went with that assumption I could use get_data from PEP 302 in order to get at the source code; get_data(source_path('module')).
But I hesistated for a long time at using get_data to fetch source code. Having get_source sitting right there was just so tempting! But then I started to consider how to handle reading bytecode. Should I have a get_bytecode method? But I then run into the problem of needing a bytecode_path method to be able to set __file__ probably for modules loaded from bytecode (and when no source is available; new semantics of 3.0). Going that route means I would have added source_path, bytecode_path, and get_bytecode just to read source and bytecode. This still doesn't deal with getting the modification time for source to see if the bytecode is stale or writing bytecode to the storage back-end through the loader.
Realizing that going the get_bytecode route duplicated functionality needlessly, I went with using get_data to read source and bytecode based on what source_path and bytecode_path return. This keeps the functionality per method simple and mostly unique. It's definitely a "misuse" of get_data as it was not meant to be used this way, but it makes sense and keeps the API simple.
With all of this put together I can provide an ABC that implements load_module in terms of the PEP 302 protocols and just a couple of other methods that handles all the stuff that is not specific to the back-end being used to store the source or bytecode. For instance, to implement a source loader, one needs to implement:
- get_data
- source_path
- is_package
- get_code
- get_source (eventually; actually figured this out just before starting this blog post)
- load_module
But the big one is when handling source and bytecode together. There you add the above methods plus:
- bytecode_path
- source_mtime
- write_bytecode
With all of this done, that leaves just two parts left to implement for the public API (get_source in terms of get_data and source_path along with a decorator I found useful). Once that is done, importlib is semantically done for CPython. I do need to talk to Jython, IronPython, and PyPy to see what they might be missing that I rely on from CPython such that if someone implements a source/bytecode loader it will still work on those VMs even if bytetcode happens to be present (this worry is thanks to PEP 370).
2009-02-02
importlib.import_module() packaged up for PyPI
Importlib is now on the Cheeseshop. I took what was in the trunk for Python 2.7 and tossed it up. At the moment I know it can go as far back as Python 2.5, but I bet it will go back even farther without issue; just need to get my hands on older interpreters to see how compatible it is.
With import_module() one does not need to call __import__ and try to comprehend how to use it. Instead you use import_module() and get a much saner API.
With import_module() one does not need to call __import__ and try to comprehend how to use it. Instead you use import_module() and get a much saner API.
2009-01-17
importlib is now in Python 3.1
[update: fixed a typo and a broken link]
Back in the summer of 2006 I interned at Google under Neal Norwitz. Part of what I did that summer was try to figure out how to potentially secure the Python interpreter for embedding into Firefox. I did finally figure out how to secure the interpreter for embedding to protect resources, culminating in a paper for a security course I took at UBC.
Part of the solution I developed required controlling import such that you couldn't import arbitrary built-in, frozen, and extension modules. As import is currently implemented, that is simply not possible to do in a secure fashion. That meant reworking or rewriting import. Since the import code was known to be a little difficult to work with I decided to rewrite it in pure Python.
Work began on October 4, 2006. At the time I was planning on making my Python security work my thesis topic with the long-term goal of making my rewrite of import the official implementation of import. Little did I know how massive of a project this would turn out to be.
Two years, three months, and 13 days later, importlib came into being for Python 3.1 in revision 68698. Between the beginning and now my security work stopped being my thesis topic (as did anything directly relating to Python), I dropped support for Python 2.x, and I learned part of the reason the C implementation is so difficult to work with is that import's semantics are rather nuanced and require juggling a lot in your head at once. Oh, and allowing different source encodings is evil.
This is easily the longest amount of time I have ever spent on a single piece of code. One of the surprising things is that the thing is not even 2,000 LOC, including tests! It just took forever to get the semantics fully backwards-compatible (short of some assumptions in the tests, you can currently run the entire test suite for Python with importlib as __import__ and have things work).
The other thing that held up checking importlib in was being too much of a perfectionist. I think I implemented importlib twice, and I still have plans on how to clean things up. Importlib has become the perfect example that your initial implementation might work, but it most definitely will not be the best implementation you can do. Heck, I still have some things to change to make the code easier to work with and more useful to users.
The perfectionist part also came out through worrying about the public API. I know people want to have access to all of the code I have written for their own importers. So I have constantly worried about how to expose it in a sane way. But API design is hard, especially when it is in Python's standard library. Get something wrong and you have to live with it for at least one extra release when you add a deprecation. This is why I am going to expose the API slowly over time and probably blog about it so that I can get feedback from people.
Now that the code is in, what are the long term plans? Well, I have notes with the code that cover what I plan to do. They start with documenting importlib.import_module. That is to be the function that everyone has asked for: a usable interface over __import__. As it stands now the interface is ``import_module(name, package)`` where 'name' is what to import, including relative imports, and 'package' is the package for the calling module. Calling the function returns the specified module, not the top module like __import__; no more fake values in fromlist! I might change the argument names, but I can't think of any other way to make it the API simpler and straight-forward.
Past that is cleaning up some things through better refactorings, exposing more code, and then exposing more of the code. But the end goal is still to get this all bootstrapped into Python 3.1 so that importlib becomes the actual implementation of __import__.
Back in the summer of 2006 I interned at Google under Neal Norwitz. Part of what I did that summer was try to figure out how to potentially secure the Python interpreter for embedding into Firefox. I did finally figure out how to secure the interpreter for embedding to protect resources, culminating in a paper for a security course I took at UBC.
Part of the solution I developed required controlling import such that you couldn't import arbitrary built-in, frozen, and extension modules. As import is currently implemented, that is simply not possible to do in a secure fashion. That meant reworking or rewriting import. Since the import code was known to be a little difficult to work with I decided to rewrite it in pure Python.
Work began on October 4, 2006. At the time I was planning on making my Python security work my thesis topic with the long-term goal of making my rewrite of import the official implementation of import. Little did I know how massive of a project this would turn out to be.
Two years, three months, and 13 days later, importlib came into being for Python 3.1 in revision 68698. Between the beginning and now my security work stopped being my thesis topic (as did anything directly relating to Python), I dropped support for Python 2.x, and I learned part of the reason the C implementation is so difficult to work with is that import's semantics are rather nuanced and require juggling a lot in your head at once. Oh, and allowing different source encodings is evil.
This is easily the longest amount of time I have ever spent on a single piece of code. One of the surprising things is that the thing is not even 2,000 LOC, including tests! It just took forever to get the semantics fully backwards-compatible (short of some assumptions in the tests, you can currently run the entire test suite for Python with importlib as __import__ and have things work).
The other thing that held up checking importlib in was being too much of a perfectionist. I think I implemented importlib twice, and I still have plans on how to clean things up. Importlib has become the perfect example that your initial implementation might work, but it most definitely will not be the best implementation you can do. Heck, I still have some things to change to make the code easier to work with and more useful to users.
The perfectionist part also came out through worrying about the public API. I know people want to have access to all of the code I have written for their own importers. So I have constantly worried about how to expose it in a sane way. But API design is hard, especially when it is in Python's standard library. Get something wrong and you have to live with it for at least one extra release when you add a deprecation. This is why I am going to expose the API slowly over time and probably blog about it so that I can get feedback from people.
Now that the code is in, what are the long term plans? Well, I have notes with the code that cover what I plan to do. They start with documenting importlib.import_module. That is to be the function that everyone has asked for: a usable interface over __import__. As it stands now the interface is ``import_module(name, package)`` where 'name' is what to import, including relative imports, and 'package' is the package for the calling module. Calling the function returns the specified module, not the top module like __import__; no more fake values in fromlist! I might change the argument names, but I can't think of any other way to make it the API simpler and straight-forward.
Past that is cleaning up some things through better refactorings, exposing more code, and then exposing more of the code. But the end goal is still to get this all bootstrapped into Python 3.1 so that importlib becomes the actual implementation of __import__.
2009-01-09
Getting importlib into the standard library by PyCon
My email to python-dev about importlib has not raised anyone's ire within 24 hours, which is a good sign. Assuming no one is going to try to stone-wall me (I would honestly be shocked if anyone did), I am committing to getting importlib into py3k by PyCon.
I have two things left before I am willing to check importlib into py3k and continue development there. One is getting the tests running under regrtest since I have not structured it for that. Second is that I have some file reorganizing in the test suite. Since svn is not exactly a lover of file renames I figure it would be best to just have that all straightened out as much as possible before I check into svn.
But PyCon is a hard deadline. No matter how much is left, the code will be checked in no later than when I get WiFi access at the hotel for PyCon.
I have two things left before I am willing to check importlib into py3k and continue development there. One is getting the tests running under regrtest since I have not structured it for that. Second is that I have some file reorganizing in the test suite. Since svn is not exactly a lover of file renames I figure it would be best to just have that all straightened out as much as possible before I check into svn.
But PyCon is a hard deadline. No matter how much is left, the code will be checked in no later than when I get WiFi access at the hotel for PyCon.
2009-01-06
The confusing terminology of imports
While I somewhat await someone to tell my importlib is broken, I am thinking about the public API I plan to expose for the package. That has put squarely in my face the issues I have with import terminology and how PEP 302 has muddled things somewhat.
PEP 302 essentially introduces the concepts of "hooks", "importers", and "loaders". When I read the PEP I come away with "hooks" being an all-encompassing term for objects that help with importing. "Importers" are are objects that define find_module and potentially load_module. "Loaders" define load_module.
The issue I have are with the definitions of "hooks" and "importers". I personally view "hooks" as things that go on sys.path_hooks, not just any object that helps with importing. For that I prefer the term "importer" as the word is tied into "import".
That means the definition of "importer", as I read the PEP, is not right to me. I prefer the term "finder" for an object that defines find_module as that is what the object does; it finds the module if possible. That would mean an importer is either a finder and/or a loader.
But how does this play out in a potential package layout? Assuming I stick with importlib as the package name (I don't want to make imp a package as that just gets messy with existing names along with what to name the existing imp and _importlib), that would mean I would want to stick all of the importers into the importlib.importers module. While that is fine, that is a lot of "import", especially if you end up with "importlib.importers.BuiltinImporter". Using a name more like importlib.hooks is easier to read and less error-prone to typing; "importlib.hooks.BuiltinImporter" has a lot less repetitiveness.
So my hope of redefining "hooks" might not work out for pragmatic reasons. Hell, PEP 302 is entitled "New Import Hooks" which seems to make it a catch-all term. And I don't want to put everything directly under importlib as that makes the namespace huge; BuiltinImporter, FrozenImporter, Finder, Loader, etc., all under the same module? I would rather have importlib.hooks have BuiltinImporter and FrozenImporter and importlib.abc has Finder, Loader, etc.
So I think I just convinced myself of having under the importlib package the hooks, abc, util, and test modules. But I am going to use the term "finder" for an object that defines find_module and thus the soon-to-exist importlib.abc.Finder ABC.
That leaves the challenge of naming the ABC for the PEP 302 protocol that covers get_source/get_code/is_package. If something defines these methods what would it be called? Is the other API an introspective one? I could go with IntrospectiveLoader or InspectLoader. I think I prefer the the former, but that's rather long and makes me feel dirty like Java makes me feel dirty with its naming. InspectLoader it is!
Obviously if any of this is nuts, please speak up. This post may seem like me just thinking out loud (and it is), but I also blog about stuff like this to get feedback from the community. While I obviously need to be happy with an API if I am going to end up having to maintain it, I want the Python community to be happy with my decisions as their will be more consumers (you) of the API than producers (me). So if you have an opinion, positive or negative, let me know (although be warned I switched off anonymous posting since OpenID is supported by Blogger and I want to cut down on WoW gold farming spam).
PEP 302 essentially introduces the concepts of "hooks", "importers", and "loaders". When I read the PEP I come away with "hooks" being an all-encompassing term for objects that help with importing. "Importers" are are objects that define find_module and potentially load_module. "Loaders" define load_module.
The issue I have are with the definitions of "hooks" and "importers". I personally view "hooks" as things that go on sys.path_hooks, not just any object that helps with importing. For that I prefer the term "importer" as the word is tied into "import".
That means the definition of "importer", as I read the PEP, is not right to me. I prefer the term "finder" for an object that defines find_module as that is what the object does; it finds the module if possible. That would mean an importer is either a finder and/or a loader.
But how does this play out in a potential package layout? Assuming I stick with importlib as the package name (I don't want to make imp a package as that just gets messy with existing names along with what to name the existing imp and _importlib), that would mean I would want to stick all of the importers into the importlib.importers module. While that is fine, that is a lot of "import", especially if you end up with "importlib.importers.BuiltinImporter". Using a name more like importlib.hooks is easier to read and less error-prone to typing; "importlib.hooks.BuiltinImporter" has a lot less repetitiveness.
So my hope of redefining "hooks" might not work out for pragmatic reasons. Hell, PEP 302 is entitled "New Import Hooks" which seems to make it a catch-all term. And I don't want to put everything directly under importlib as that makes the namespace huge; BuiltinImporter, FrozenImporter, Finder, Loader, etc., all under the same module? I would rather have importlib.hooks have BuiltinImporter and FrozenImporter and importlib.abc has Finder, Loader, etc.
So I think I just convinced myself of having under the importlib package the hooks, abc, util, and test modules. But I am going to use the term "finder" for an object that defines find_module and thus the soon-to-exist importlib.abc.Finder ABC.
That leaves the challenge of naming the ABC for the PEP 302 protocol that covers get_source/get_code/is_package. If something defines these methods what would it be called? Is the other API an introspective one? I could go with IntrospectiveLoader or InspectLoader. I think I prefer the the former, but that's rather long and makes me feel dirty like Java makes me feel dirty with its naming. InspectLoader it is!
Obviously if any of this is nuts, please speak up. This post may seem like me just thinking out loud (and it is), but I also blog about stuff like this to get feedback from the community. While I obviously need to be happy with an API if I am going to end up having to maintain it, I want the Python community to be happy with my decisions as their will be more consumers (you) of the API than producers (me). So if you have an opinion, positive or negative, let me know (although be warned I switched off anonymous posting since OpenID is supported by Blogger and I want to cut down on WoW gold farming spam).
2009-01-03
importlib hits beta (with PEP 263 support!)
As I type this I am doing a ``bzr push`` to importlib that puts the code at beta quality. At this point I pass all of my own tests and the failures I have from the standard library are either from explicit checking of exception messages or from code not expecting __loader__ to be defined.
This means I finally dealt with how to read source code that has a defined encoding. Supporting PEP 263 has literally taken me nearly a year to support. "Why did it take so long?", you might ask. Well, this is where the difference between 'compile' and what import does becomes glaringly apparent and Python 3.0 shows how badly even the interpreter intermixed strings and bytes and strings as text.
When you want to create a code object, you use 'compile'. Typically you simply take a string, pass it in with the proper arguments, and out comes a code object. But those pesky encoding declarations make things a wee bit tricky. You see, in Python 2.x, when you passed in a string, you could pass in a string read from a file raw with universal newline support. You didn't have to decode anything or worry about newlines since the former was handled by the parser and the latter was handled by 'open'. This meant the parser simply took the string's buffer and used that as input to tokenize everything based on either the default encoding for source code or the encoding declaration.
Enter Python 3.0 and the distinction between strings and bytes. Passing in a string means that the underlying buffer is already decoded for you. But you see, the parser doesn't know that. Instead it just sees that it was given a 'const char *' and figures it needs to decode it. Now if you have no encoding declaration this works fine as it is assumed to be encoded using the default encoding for strings. But what if you have an encoding declaration for the source? Well, turns out you are hosed since the tokenizer has no idea that the 'const char *' it is working with is coming from a decoded string and not some encoded bytes. Pass 'compile' a string which is encoded underneath as UTF-8 but which contains a Latin-1 encoding declaration for the source and the tokenizer throws a decoding hissy fit.
What is a core developer to do? He created an issue for it of course! Issue 4626 covers the problem of 'compile' not playing well with a string that has a declared encoding other than UTF-8. I spent most of my day trying to fix this bug. It turns out that 'compile' underneath it all simply gets the buffer for its first argument if it is a bytes or string object and passes it along to be parsed. It seems like a waste to throw out the knowledge that the 'const char *' that is being parsed is already decoded as UTF-8, so I tried to pass that information around.
But I hit my first snag. Since this is C I can't simply toss in a new argument to a function and have everything just work. Not only does that break any other code that calls the function I added an argument to, it breaks ABI and API compatibility for extension authors, where the latter is an absolute no-no in Python and the latter is just really bad.
OK, so I try a hacked solution by adding a flag to the compiler flags that is meant more for __future__ statements. It really isn't that bad since it is already used for other backwards-compatibility stuff, so I don't feel dirty. But then I discover that the flags used by the compiler are not the same used by the parser, even when they do overlap. Oops. So I find the proper macro and tweak it to translate from compiler flags to parser flags. That way I can send down the call chain into the parser that the 'const char *' it is working off of is already decoded in the default encoding.
That is when I hit the next snag. Turns out that the parser is not compiled with the various Python objects linked in. That means calls into the PyUnicode API are not supported. And keeping the parser simple is on purpose. Suddenly linking in the various Python objects would make it tempting to walk away from this. This also means I can't call PyUnicode_GetDefaultEncoding to know what the bytes are encoded in.
So I take a different tack and try to explicitly translate to UTF-8. That way the flag I have added simply means that it is all already translated explicitly into UTF-8. But even that doesn't work easily. After battling with pgen/not-pgen code in the parser and everything else, I decide to take another approach.
Have I mentioned I don't like working on the parser? I didn't like mucking with it for doing the parse tree to AST conversion code, I didn't like mucking with it to fix the bug that I blogged about a couple months ago, and I didn't like having to deal with it today.
Time for a different approach. Remember how earlier in this post I mentioned universal newline support along with encoding being a thorn in my side? The reason I mentioned newlines is because passing in a bytes object read directly from the file into 'compile' gets around the encoding mess as the parser can just go ahead and do its own decoding since bytes have not been decoded ahead of time. But reading bytes from a file doesn't give you universal newlines support. Turns out that the universal newline support that the parser uses is entirely based on using FILE pointers and the requisite C functions. Thus issue 4628 was born.
Initially I thought I was going to have to write some C code to add to the bytes object to have bytes.splitlines do proper universal newlines (as it stands now that method just splits on \n, \r, and \r\n all in the same string). But then I realized that the basic algorithm is dead-simple in Python code and I had already abstracted out the reading of source code thanks to PEP 302 protocols and the get_source method. Running with my realization, I wrote the code to read in the bytes, figure out the newlines used, and do a bytes.replace to make the newlines universal. Simple.
But this is where the whole issue of compatibility and performance comes into play. Consider compatbility with get_source. That is supposed to return a decoded string according to PEP 302. That's fine, but that sucks in my case as I want bytes. Do I define a new method like get_bytes or add an argument to get_source like 'raw' that flags I want just the bytes? Or do I just not care and tweak the definition of get_source to return anything that 'compile' will handle properly? And if I have it return bytes, does it have to have universal newlines, or should import do that?
And this all then spills into performance. Compare having get_source return a string instead of bytes. To return a decoded string takes two file system calls; you have to read the first two files to find the possible source encoding and then open the file again with the proper encoding specified so that you properly decode the file. Luckily universal newline support is done in a streaming manner so there is not real penalty for that.
Using bytes cuts those stat calls down to one as you only need to read the file. But the memory pressure potentially rises as you need to handle the universal newline support after the file has been completely read.
Considering how I/O bound importing is, I went with the latter case of more memory but less stat calls. It's bad enough that importing from the file system potentially takes four stat calls per entry on sys.path/__path__ just to figure out what file to use for the import, let alone actually reading the files and dealing with ImportWarning for directories that match by name but lack a __init__.py file. But then what about the potential need to tweak the API for loaders to return the bytes for source?
There the universal newlines overhead comes into play. If the method that returns the bytes for source code handles the universal newlines support, that's annoying as that will be a common thing to have to do. It would be easier to simply do the universal newline translation in import itself so that loader developers do not need to deal with this themselves.
But what about those loaders that take the time to actually do the newline translation upfront? If I decide to create a sqlite loader that guarantees that the stored source code already has its newlines translated, why should the loader pay the penalty of me trying to figure out what newlines are used?
Luckily the penalty for detecting that universal newlines is already being used is negligble as it is simply iterating through the bytes until the first possible newline is found. I will probably just shift it out of get_source and just have import do it automatically.
Yes, this is the stuff I think about when contemplating what the API will be for importlib in the end. I have put in enough time into this code that I don't want to screw up the API and have to support something that just frustrates me for the rest of my days.
Where does this leave importlib? At this moment I consider it semantically complete short of someone finding an incompatibility. It passes on OS X for me, although I don't know about Windows as I may have made some silly path separator assumption (if anyone runs importlib's test suite by doing ``python3.0 -m tests.__init__``, let me know if things pass for you). Assuming no bugs are found that leaves removing Python 2.x cruft in the code and cleaning up the docstrings. After all that is done I will create The Great Import Function where you can do something like ``importlib.import_module('some.module')`` and have it give you back 'some.module'. Then I will try to get the code into Python 3.1 for at least inclusion in the standard library, if not bootstrapped in as the official implementation of import.
This means I finally dealt with how to read source code that has a defined encoding. Supporting PEP 263 has literally taken me nearly a year to support. "Why did it take so long?", you might ask. Well, this is where the difference between 'compile' and what import does becomes glaringly apparent and Python 3.0 shows how badly even the interpreter intermixed strings and bytes and strings as text.
When you want to create a code object, you use 'compile'. Typically you simply take a string, pass it in with the proper arguments, and out comes a code object. But those pesky encoding declarations make things a wee bit tricky. You see, in Python 2.x, when you passed in a string, you could pass in a string read from a file raw with universal newline support. You didn't have to decode anything or worry about newlines since the former was handled by the parser and the latter was handled by 'open'. This meant the parser simply took the string's buffer and used that as input to tokenize everything based on either the default encoding for source code or the encoding declaration.
Enter Python 3.0 and the distinction between strings and bytes. Passing in a string means that the underlying buffer is already decoded for you. But you see, the parser doesn't know that. Instead it just sees that it was given a 'const char *' and figures it needs to decode it. Now if you have no encoding declaration this works fine as it is assumed to be encoded using the default encoding for strings. But what if you have an encoding declaration for the source? Well, turns out you are hosed since the tokenizer has no idea that the 'const char *' it is working with is coming from a decoded string and not some encoded bytes. Pass 'compile' a string which is encoded underneath as UTF-8 but which contains a Latin-1 encoding declaration for the source and the tokenizer throws a decoding hissy fit.
What is a core developer to do? He created an issue for it of course! Issue 4626 covers the problem of 'compile' not playing well with a string that has a declared encoding other than UTF-8. I spent most of my day trying to fix this bug. It turns out that 'compile' underneath it all simply gets the buffer for its first argument if it is a bytes or string object and passes it along to be parsed. It seems like a waste to throw out the knowledge that the 'const char *' that is being parsed is already decoded as UTF-8, so I tried to pass that information around.
But I hit my first snag. Since this is C I can't simply toss in a new argument to a function and have everything just work. Not only does that break any other code that calls the function I added an argument to, it breaks ABI and API compatibility for extension authors, where the latter is an absolute no-no in Python and the latter is just really bad.
OK, so I try a hacked solution by adding a flag to the compiler flags that is meant more for __future__ statements. It really isn't that bad since it is already used for other backwards-compatibility stuff, so I don't feel dirty. But then I discover that the flags used by the compiler are not the same used by the parser, even when they do overlap. Oops. So I find the proper macro and tweak it to translate from compiler flags to parser flags. That way I can send down the call chain into the parser that the 'const char *' it is working off of is already decoded in the default encoding.
That is when I hit the next snag. Turns out that the parser is not compiled with the various Python objects linked in. That means calls into the PyUnicode API are not supported. And keeping the parser simple is on purpose. Suddenly linking in the various Python objects would make it tempting to walk away from this. This also means I can't call PyUnicode_GetDefaultEncoding to know what the bytes are encoded in.
So I take a different tack and try to explicitly translate to UTF-8. That way the flag I have added simply means that it is all already translated explicitly into UTF-8. But even that doesn't work easily. After battling with pgen/not-pgen code in the parser and everything else, I decide to take another approach.
Have I mentioned I don't like working on the parser? I didn't like mucking with it for doing the parse tree to AST conversion code, I didn't like mucking with it to fix the bug that I blogged about a couple months ago, and I didn't like having to deal with it today.
Time for a different approach. Remember how earlier in this post I mentioned universal newline support along with encoding being a thorn in my side? The reason I mentioned newlines is because passing in a bytes object read directly from the file into 'compile' gets around the encoding mess as the parser can just go ahead and do its own decoding since bytes have not been decoded ahead of time. But reading bytes from a file doesn't give you universal newlines support. Turns out that the universal newline support that the parser uses is entirely based on using FILE pointers and the requisite C functions. Thus issue 4628 was born.
Initially I thought I was going to have to write some C code to add to the bytes object to have bytes.splitlines do proper universal newlines (as it stands now that method just splits on \n, \r, and \r\n all in the same string). But then I realized that the basic algorithm is dead-simple in Python code and I had already abstracted out the reading of source code thanks to PEP 302 protocols and the get_source method. Running with my realization, I wrote the code to read in the bytes, figure out the newlines used, and do a bytes.replace to make the newlines universal. Simple.
But this is where the whole issue of compatibility and performance comes into play. Consider compatbility with get_source. That is supposed to return a decoded string according to PEP 302. That's fine, but that sucks in my case as I want bytes. Do I define a new method like get_bytes or add an argument to get_source like 'raw' that flags I want just the bytes? Or do I just not care and tweak the definition of get_source to return anything that 'compile' will handle properly? And if I have it return bytes, does it have to have universal newlines, or should import do that?
And this all then spills into performance. Compare having get_source return a string instead of bytes. To return a decoded string takes two file system calls; you have to read the first two files to find the possible source encoding and then open the file again with the proper encoding specified so that you properly decode the file. Luckily universal newline support is done in a streaming manner so there is not real penalty for that.
Using bytes cuts those stat calls down to one as you only need to read the file. But the memory pressure potentially rises as you need to handle the universal newline support after the file has been completely read.
Considering how I/O bound importing is, I went with the latter case of more memory but less stat calls. It's bad enough that importing from the file system potentially takes four stat calls per entry on sys.path/__path__ just to figure out what file to use for the import, let alone actually reading the files and dealing with ImportWarning for directories that match by name but lack a __init__.py file. But then what about the potential need to tweak the API for loaders to return the bytes for source?
There the universal newlines overhead comes into play. If the method that returns the bytes for source code handles the universal newlines support, that's annoying as that will be a common thing to have to do. It would be easier to simply do the universal newline translation in import itself so that loader developers do not need to deal with this themselves.
But what about those loaders that take the time to actually do the newline translation upfront? If I decide to create a sqlite loader that guarantees that the stored source code already has its newlines translated, why should the loader pay the penalty of me trying to figure out what newlines are used?
Luckily the penalty for detecting that universal newlines is already being used is negligble as it is simply iterating through the bytes until the first possible newline is found. I will probably just shift it out of get_source and just have import do it automatically.
Yes, this is the stuff I think about when contemplating what the API will be for importlib in the end. I have put in enough time into this code that I don't want to screw up the API and have to support something that just frustrates me for the rest of my days.
Where does this leave importlib? At this moment I consider it semantically complete short of someone finding an incompatibility. It passes on OS X for me, although I don't know about Windows as I may have made some silly path separator assumption (if anyone runs importlib's test suite by doing ``python3.0 -m tests.__init__``, let me know if things pass for you). Assuming no bugs are found that leaves removing Python 2.x cruft in the code and cleaning up the docstrings. After all that is done I will create The Great Import Function where you can do something like ``importlib.import_module('some.module')`` and have it give you back 'some.module'. Then I will try to get the code into Python 3.1 for at least inclusion in the standard library, if not bootstrapped in as the official implementation of import.
2008-12-24
importlib hits alpha
As of now I am willing to declare importlib (bzr repository) an alpha quality replacement for __import__. Do note that importlib is 3.0 code as that is what I test against. There is also probably some 3.0-specific code as well, but I specifically can't think of any off the top of my head short of some print statements from the test driver.
Currently there are two things I need to fix. One is what exceptions get thrown by importlib. For instance, if some code cannot be decoded as UTF-8 then __import__ raises SyntaxError. Importlib, on the other hand, raises UnicodeDecodeError. Same goes for when a parent package cannot be imported for for submodule; __import__ raises SystemError while importlib raises ImportError. Rather minor stuff that just requires some extra code to raise the proper exception.
The second reason is entirely the fault of the 'compile' built-in. It turns out that 'compile' just doesn't do as much work as __import__ does when processing a string or bytes object. In the case of strings, if you pass in code that has an encoding declared then 'compile' actually tries to decode the string since it works off the buffer interface to the string. Well that doesn't work since the string has already been decoded to UTF-8 by the time I give it to 'compile'. This is reported as issue 4626. Probably the best solution would be to change it so that if a string is passed in the decoding step is skipped.
The issue with bytes is universal newline support. Turns out 'compile' will decode a bytes object properly, encoding declaration and all, but it won't support universal newlines as that feature completely relies on fgets on files at the C level. This is reported as issue 4628. To solve this universal newlines needs to be supported for bytes somehow.
Unfortunately solving either issue is not necessarily simple as this dives into the interface with the parser. That chunk of code tends to only do its fancy tricks when it is working with files. It was painful enough solving the last bug I found in the parser that I do not particularly want to hack on it again. This means I am only going to solve one of the issues. And while issue 4626 is probably the easier of the two to solve, I would rather see if I can get issue 4628 fixed as that cuts down on stat calls and has a better chance of being more useful outside of my use case (although it potentially has a higher memory cost).
It is one of my New Years resolutions for next year to get importlib finished up to the point that it is ready to act as a replacement for __import__. Currently that means ignoring how to expose the code publicly beyond that of __import__. This will not be the case forever, but I have some refactorings I want to do that might tweak the APIs somewhat. And since any API that becomes public needs to be supported for a while I do not want to rush this. All of this is spelled out in the NOTES file for those of you who are curious what the exact plans are and the rough order I plan to address them.
Currently there are two things I need to fix. One is what exceptions get thrown by importlib. For instance, if some code cannot be decoded as UTF-8 then __import__ raises SyntaxError. Importlib, on the other hand, raises UnicodeDecodeError. Same goes for when a parent package cannot be imported for for submodule; __import__ raises SystemError while importlib raises ImportError. Rather minor stuff that just requires some extra code to raise the proper exception.
The second reason is entirely the fault of the 'compile' built-in. It turns out that 'compile' just doesn't do as much work as __import__ does when processing a string or bytes object. In the case of strings, if you pass in code that has an encoding declared then 'compile' actually tries to decode the string since it works off the buffer interface to the string. Well that doesn't work since the string has already been decoded to UTF-8 by the time I give it to 'compile'. This is reported as issue 4626. Probably the best solution would be to change it so that if a string is passed in the decoding step is skipped.
The issue with bytes is universal newline support. Turns out 'compile' will decode a bytes object properly, encoding declaration and all, but it won't support universal newlines as that feature completely relies on fgets on files at the C level. This is reported as issue 4628. To solve this universal newlines needs to be supported for bytes somehow.
Unfortunately solving either issue is not necessarily simple as this dives into the interface with the parser. That chunk of code tends to only do its fancy tricks when it is working with files. It was painful enough solving the last bug I found in the parser that I do not particularly want to hack on it again. This means I am only going to solve one of the issues. And while issue 4626 is probably the easier of the two to solve, I would rather see if I can get issue 4628 fixed as that cuts down on stat calls and has a better chance of being more useful outside of my use case (although it potentially has a higher memory cost).
It is one of my New Years resolutions for next year to get importlib finished up to the point that it is ready to act as a replacement for __import__. Currently that means ignoring how to expose the code publicly beyond that of __import__. This will not be the case forever, but I have some refactorings I want to do that might tweak the APIs somewhat. And since any API that becomes public needs to be supported for a while I do not want to rush this. All of this is spelled out in the NOTES file for those of you who are curious what the exact plans are and the rough order I plan to address them.
2008-08-18
Found out why my bootstrapping of importlib has been failing (probably)
In my last update on importlib nearly two months ago, I mentioned I had three failing tests; two were seemingly BOM related while the latter was just a lack of a feature. The feature got implemented, leaving just the BOM issues.
Having a failure seem to be related to the BOM made me think that some how I was not handling the encoding of source files properly when they defined a specific encoding. I thought that somehow the parser handled things differently enough between reading from files and using a string that I was being prevented from reading from a file directly and feeding that info to compile().
Well, today I decided to dig into the issue after having done some other compile() work yesterday. Turns out that the problem was not in any of my code, but the parser's API for finding out the encoding of a source file. If you look at Parser/tokenizer.c:PyTokenizer_FindEncoding() you will discover that at some point open() is called, but with a path of NULL. Oops.
And because of the one place where PyTokenizer_FindEncoding() didn't use PyErr_Occurred() to see if an error occurred, the exception went unnoticed. I bet at some point PyErr_Clear() is blindly called and that hid the exception.
But in my bootstrapping of importlib, the call path is not the same, and at some point PyErr_Occurred() is checked and the exception exposes itself in an odd place. Anyway, a few hours with gdb and I figured all of this out.
So it's quite possible that my bootstrapping of importlib is actually ready to go as soon as this pre-existing bug is fixed!
Having a failure seem to be related to the BOM made me think that some how I was not handling the encoding of source files properly when they defined a specific encoding. I thought that somehow the parser handled things differently enough between reading from files and using a string that I was being prevented from reading from a file directly and feeding that info to compile().
Well, today I decided to dig into the issue after having done some other compile() work yesterday. Turns out that the problem was not in any of my code, but the parser's API for finding out the encoding of a source file. If you look at Parser/tokenizer.c:PyTokenizer_FindEncoding() you will discover that at some point open() is called, but with a path of NULL. Oops.
And because of the one place where PyTokenizer_FindEncoding() didn't use PyErr_Occurred() to see if an error occurred, the exception went unnoticed. I bet at some point PyErr_Clear() is blindly called and that hid the exception.
But in my bootstrapping of importlib, the call path is not the same, and at some point PyErr_Occurred() is checked and the exception exposes itself in an odd place. Anyway, a few hours with gdb and I figured all of this out.
So it's quite possible that my bootstrapping of importlib is actually ready to go as soon as this pre-existing bug is fixed!
2008-06-21
Update on bootstrapping importlib for Python 3.1
I just updated my py3k-importlib bzr branch so that I was running on post 3.0b1 code. At the moment I have three failing tests: test_coding, test_pep263, and test_runpy. The first two are BOM errors (although test_pep263 only fails when run through regrtest). test_runpy fails because I have not implemented __package__ support yet (although I have diagrammed how to properly set the attribute). I hoping that the BOM failure, whatever the cause, once fixed will solve both failures. And the test_runpy failure is just a matter of me adding in __package__ support which should not be too difficult.
I really should be worrying more about finishing up PEP 3108. But I must admit, hacking on my own stuff is a lot more fun. =) Although I am feeling more and more like I need to rework the testing suite to use doctest and be more blackbox and less whitebox. Weighing in at 2507 lines (according to 'wc'), the testing suite is rather large and has grown organically, so it is not the most organized chunk of code in the world.
I really should be worrying more about finishing up PEP 3108. But I must admit, hacking on my own stuff is a lot more fun. =) Although I am feeling more and more like I need to rework the testing suite to use doctest and be more blackbox and less whitebox. Weighing in at 2507 lines (according to 'wc'), the testing suite is rather large and has grown organically, so it is not the most organized chunk of code in the world.
2008-04-22
One down, three to go
I managed to knock off another failing test for integrating importlib into Py3K; test_pkg. That leaves three: test_coding, test_pep263, and test_runpy. The first two have something to do with the BOM and probably how I am trying to figure out the encoding of the source file. Test_runpy has something to do with relative imports. Hopefully solving test_coding or test_pep263 will knock the other one off, meaning I only have two bugs left to deal with by June 1.
2008-03-08
import, illustrated, and PyCon registration stats
This past Tuesday I gave my import talk to VanPyZ, my local Python user's group. Turns out it was not only long, I had details in there that were not what people cared about (specifically covering the details of choosing what module to have __import__ return). I also realized that explaining import in text was not as clear as I thought it was (problem of having been neck deep in this stuff for so long).
And so, thanks to my supervisor paying for a copy of OmniGraffle Pro, I have been reworked my presentation using flowcharts. I think the charts make the presentation much easier to follow as now you can visually follow the various cases that must be dealt with instead of reading some pseudocode or me trying to explain things through bullet points.
A side-effect of doing multiple flowcharts on import for my presentation is that I wanted to make a single, comprehensive flowchart of what import does. That flowchart, along with the master graffle file, is in svn. As I said on Jaiku, the algorithm doesn't look complicated when drawn as a flowchart, but boy is it long!
I will probably eventually finish the flowchart to be comprehensive. As of right now it lacks the details of deciding what module to return. I also left out any details on how the source/bytecode importer/loader works and I would like to get that drawn as well.
And in PyCon news, it looks like there will be 1029 attendees at PyCon if everyone who still needs to pay does. Possibly over a thousand attendees! We had over 600 last year. The growth is just phenomenal. I still remember the first PyCon when it was about 200 attendees and we had A/C issues with all the rooms. Now we have sponsored sprints, a keynote each day, and paid lunches. The conference has come a long way.
And so, thanks to my supervisor paying for a copy of OmniGraffle Pro, I have been reworked my presentation using flowcharts. I think the charts make the presentation much easier to follow as now you can visually follow the various cases that must be dealt with instead of reading some pseudocode or me trying to explain things through bullet points.
A side-effect of doing multiple flowcharts on import for my presentation is that I wanted to make a single, comprehensive flowchart of what import does. That flowchart, along with the master graffle file, is in svn. As I said on Jaiku, the algorithm doesn't look complicated when drawn as a flowchart, but boy is it long!
I will probably eventually finish the flowchart to be comprehensive. As of right now it lacks the details of deciding what module to return. I also left out any details on how the source/bytecode importer/loader works and I would like to get that drawn as well.
And in PyCon news, it looks like there will be 1029 attendees at PyCon if everyone who still needs to pay does. Possibly over a thousand attendees! We had over 600 last year. The growth is just phenomenal. I still remember the first PyCon when it was about 200 attendees and we had A/C issues with all the rooms. Now we have sponsored sprints, a keynote each day, and paid lunches. The conference has come a long way.
2007-10-30
importlib version of zipimport is finished
I finally got around to bothering to tweak test_zipimport to run using my importlib implementation of zipimport. My implementation is not fully backwards-compatible, but the tests that fail are because the current version was written in C. I mean who cares if the 'prefix' attribute on a zipimport instance doesn't end with a path separator? You should be using os.path functions anyway which handles that. And there is nothing wrong if someone names a zip file 'None', and thus makes None itself a legitimate path argument.
So I consider it done at this point. Only took 228 lines of Python code to pull off. Compare that to 1197 lines of C and I think that's an improvement. =) My implementation could even be improved to allow for writing back out bytecode when it is missing.
As for importlib, I have PyCon stuff and coding stuff. For PyCon, I have submitted for a 30 minute talk on the algorithm of import. I am also working on a tutorial proposal on imports. That is looking to have:
As for coding, with this out of the way I am now working on getting a C implementation of the core parts of the 'warnings' module done. That will let me do actual testing under Py3K to look for backwards-compatibility problems. And then I begin ripping out C code and re-implementing C APIs.
So much to do. But I also have demo and paper submission deadlines for my Ph.D. research that take precedence so I have no idea when I am going to any of this.
So I consider it done at this point. Only took 228 lines of Python code to pull off. Compare that to 1197 lines of C and I think that's an improvement. =) My implementation could even be improved to allow for writing back out bytecode when it is missing.
As for importlib, I have PyCon stuff and coding stuff. For PyCon, I have submitted for a 30 minute talk on the algorithm of import. I am also working on a tutorial proposal on imports. That is looking to have:
- A longer version of my PyCon talk with 2.x details included (my PyCon talk is 3.0 only as it keeps the details simpler).
- How importing source, bytecode, extension modules, and frozen modules work.
- PEP 302 optional extensions and why you should care.
- How to make extension modules reload safely (that one is for Doug, and for 2.x the answer is you don't =).
- How to make your package load a custom importer/loader automatically.
- Importlib so that you don't have to re-implement the basics.
- Shutting down bytcode generation using importlib.
- Creation of an HTTP-based importer/loader.
- Going over a real-life example using the zipimport re-implementation.
As for coding, with this out of the way I am now working on getting a C implementation of the core parts of the 'warnings' module done. That will let me do actual testing under Py3K to look for backwards-compatibility problems. And then I begin ripping out C code and re-implementing C APIs.
So much to do. But I also have demo and paper submission deadlines for my Ph.D. research that take precedence so I have no idea when I am going to any of this.
2007-10-13
Importlib update
Importlib has now been (roughly) bootstrapped into Py3K in my py3k-importlib branch. There are some tests that are still failing that I have not tried to fix yet. I also have not removed any C code yet so there is still a chance of some dependency that I don't know about yet.
And there is still the issue of 'warnings' not being a built-in module. Neal Norwitz wrote an initial C version of the critical stuff, but it isn't complete. The biggest issue is that it doesn't pick up anything that is set on 'warnings' itself (e.g., 'filters', 'showwarnings'). That's bad as various attributes in 'warnings' get set externally frequently. My proposed solution is to see if 'warnings' has been imported, and if so use the attributes from there, otherwise fall back on internal C stuff. That way the module is entirely independent of Python code and thus doesn't cause me any bootstrap issues.
Once 'warnings' is built-in I will be able to rip out C code to make sure this whole thing actually works. Then I can fix any bugs I have for importlib itself. That should be it at that point and I should hopefully be able to move my work into Py3K itself.
And there is still the issue of 'warnings' not being a built-in module. Neal Norwitz wrote an initial C version of the critical stuff, but it isn't complete. The biggest issue is that it doesn't pick up anything that is set on 'warnings' itself (e.g., 'filters', 'showwarnings'). That's bad as various attributes in 'warnings' get set externally frequently. My proposed solution is to see if 'warnings' has been imported, and if so use the attributes from there, otherwise fall back on internal C stuff. That way the module is entirely independent of Python code and thus doesn't cause me any bootstrap issues.
Once 'warnings' is built-in I will be able to rip out C code to make sure this whole thing actually works. Then I can fix any bugs I have for importlib itself. That should be it at that point and I should hopefully be able to move my work into Py3K itself.
2007-09-16
importlib ported to Py3K
I just finished porting importlib over to Py3K. I basically created a Py3K directory in importlib's sandbox and just copied the requisite files over, ran them through 2to3, and then made changes as needed (first in the original files when possible, and then Py3K-only changes).
The only Py3K-specific changes I had to make was for the bytes type. Stuff such as imp.get_magic() and reading bytecode in binary returns bytes and that just doesn't work for comparing against string literals or passing to ord(). =) I also had some integer division that should have been flooring instead of true division. But otherwise it's now converted.
Unfortunately if I make any changes to the original files I am going to have to forward-port it. What I think would be the best solution is have a script that copies the originals, run them through 2to3, and then apply Py3K-specific patches (which would be checked into svn). That way I can develop on the 2.x code and then just make the Py3K code an after-thought. If I have to tweak any code in the 2.x version I will probably bother with this, but not right now. =)
The only Py3K-specific changes I had to make was for the bytes type. Stuff such as imp.get_magic() and reading bytecode in binary returns bytes and that just doesn't work for comparing against string literals or passing to ord(). =) I also had some integer division that should have been flooring instead of true division. But otherwise it's now converted.
Unfortunately if I make any changes to the original files I am going to have to forward-port it. What I think would be the best solution is have a script that copies the originals, run them through 2to3, and then apply Py3K-specific patches (which would be checked into svn). That way I can develop on the 2.x code and then just make the Py3K code an after-thought. If I have to tweak any code in the 2.x version I will probably bother with this, but not right now. =)
2007-08-27
zipimport rewrite is done (I think)
My pure Python rewrite of zipimport is basically complete, sort of. I tried running against the zipimport tests in 2.6 but I run into some funky issues like the first entry on sys.path not being passed to the path hook (these tests are not using importlib for import).
Anyway, as best as I can tell from the tests that I wrote from scratch to test the individual methods the thing works. I might try to get the tests from 2.6 working at some point, but not right now. =)
Anyway, as best as I can tell from the tests that I wrote from scratch to test the individual methods the thing works. I might try to get the tests from 2.6 working at some point, but not right now. =)
2007-08-23
Importlib is passing all tests (again)
I now consider the simplification refactoring done in importlib. It is passing all of its own unit tests on top of working as the 2.6 importer and passing all of the 2.6 regression tests.
I have now started a text file that lists what is left to be done to get importlib into Py3K as the One True Import. Next on the list, the zipimport rewrite that triggered this refactoring job. =)
I have now started a text file that lists what is left to be done to get importlib into Py3K as the One True Import. Next on the list, the zipimport rewrite that triggered this refactoring job. =)
2007-08-21
Importlib ... almost ... there ...
The code refactoring is now done for importlib. Now I am trying to make it pass Python's entire regression test suite as the import implementation. This has led to the discovery that runpy (the module that implements the '-m' command-line option) wants loaders to be stateless. That kind of sucks as that will spike the amount of stat calls I will need to make, but that's life when you are dealing with a filesystem and you don't assume its immutable. =)
And I have now been given the task of getting importlib into Py3K. I won't make a1, but since I am not introducing new features that is not an issue. But I need to get the 2.6 version squared away first.
And I have now been given the task of getting importlib into Py3K. I won't make a1, but since I am not introducing new features that is not an issue. But I need to get the 2.6 version squared away first.
2007-08-09
Making it easier to write custom code formats while doing the py/pyc dance
Wouldn't it be nice if you could write Python code with slightly tweaked syntax or something but avoid having to write all the import code to pull it off? Wouldn't you like to have bytecode compilation supported as well?
As an example, consider Quixote. It has a syntax format called PTL where string literals in Python code end up being used as output. They had to write a custom __import__ implementation to get proper support for imports of PTL files. And there is no bytecode support.
When I was designing importlib I wanted to make sure that situations like PTL could be supported without jumping through ridiculous hoops like completely reimplementing entire parts of the import machinery. I figured there should be a way to allow for a class to be written that acted as a delegate between the file system and the code that managed the source/bytecode dance.
That is when I came up with the idea of handlers. From PEP 302, importers are for querying a backend as to whether it has the desired module and loaders are for actually get the source or bytecode from the backend store. Handlers have been designed to handle the data received from the loader and to make requests of the loader as needed. Thus the py/pyc handler deals with validating the bytecode is new enough and has the proper magic cookie, requesting source when needed, and creating new bytecode to be stored by the loader.
And to support this I came up with an API for loaders to implement to be used by the py/pyc handler. But last night, when I decided to stop working in my zipimport rewrite, I looked at the API and realized it was not simple enough. I realized that the py/pyc dance requires some information up front while some other info is optional and completely depends on whether source and/or bytecode is available for a module. I also realized that loaders can cache any info they needed and thus I did not need to complicate things by passing around opaque token objects through the handler to give back to the loader to minimize wasted lookup costs and such.
Let's look at what it takes to import a source or bytecode module. To begin with, you need to know the values for __name__, __file__, __loader__ (optional, but since this is a PEP 302 implementation its easy to provide), and __path__ if dealing with a package. This all needs to be set on a module before any code is executed.
You will also need to know very quickly whether source and/or bytecode is available for the named module. Based on that you branch on what information you need. But the available formats or the module and the attributes that must be set before executed code are all upfront costs. So I figured those should be passed in to the handler at call time and thus not require the loader to implement as part of any API.
After that what is needed varies based on the formats the module is available in. If there is bytecode you need to get said bytecode. If there is both bytecode and source you need the modification time of the source to see if the bytecode is stale. If there is only source or the bytecode is stale you need to be able to get the source itself. Finally, if the bytecode turned out to be bad (stale because its out of date or bad bytecode cookie) you need to be able to write out the new bytecode. So that's four things that the loader needs an API for: read bytecode, read source, source modification time, and writing out new bytecode.
Now the question becomes how should this API look. Originally, as I mentioned above, there was an idea of opaque token objects that represented paths that were passed around within the handler code and given back to the loader. This was to make the loader stateless and to minimize any repeated lookups on the filesystem (or whatever store the loader used).
But then I realized the loader did not need to be stateless for any reason. Plus if someone who implemented a loader wanted to optimize they could easily cache information in the loader themselves. That meant the API really only needed the module's name. That does add the possible inconvenience of verifying that a request was reasonable for that specific module (e.g., requesting the bytecode for a module who has no bytecode yet), but that should be minimal. And the biggest perk of just requiring the module's name is that PEP 302's optional extensions for importers and loaders can be used (the get_source method).
So that is the direction I am going with for my API change to importlib. I am also breaking up the filesystem importer and loader I wrote and having extension modules and py/pyc modules have their own importers and loaders. That way the generalization does not need to be so severe on everything. Plus extension modules are just plain different from py/pyc files and trying to make them work with the same API seems extreme.
As an example, consider Quixote. It has a syntax format called PTL where string literals in Python code end up being used as output. They had to write a custom __import__ implementation to get proper support for imports of PTL files. And there is no bytecode support.
When I was designing importlib I wanted to make sure that situations like PTL could be supported without jumping through ridiculous hoops like completely reimplementing entire parts of the import machinery. I figured there should be a way to allow for a class to be written that acted as a delegate between the file system and the code that managed the source/bytecode dance.
That is when I came up with the idea of handlers. From PEP 302, importers are for querying a backend as to whether it has the desired module and loaders are for actually get the source or bytecode from the backend store. Handlers have been designed to handle the data received from the loader and to make requests of the loader as needed. Thus the py/pyc handler deals with validating the bytecode is new enough and has the proper magic cookie, requesting source when needed, and creating new bytecode to be stored by the loader.
And to support this I came up with an API for loaders to implement to be used by the py/pyc handler. But last night, when I decided to stop working in my zipimport rewrite, I looked at the API and realized it was not simple enough. I realized that the py/pyc dance requires some information up front while some other info is optional and completely depends on whether source and/or bytecode is available for a module. I also realized that loaders can cache any info they needed and thus I did not need to complicate things by passing around opaque token objects through the handler to give back to the loader to minimize wasted lookup costs and such.
Let's look at what it takes to import a source or bytecode module. To begin with, you need to know the values for __name__, __file__, __loader__ (optional, but since this is a PEP 302 implementation its easy to provide), and __path__ if dealing with a package. This all needs to be set on a module before any code is executed.
You will also need to know very quickly whether source and/or bytecode is available for the named module. Based on that you branch on what information you need. But the available formats or the module and the attributes that must be set before executed code are all upfront costs. So I figured those should be passed in to the handler at call time and thus not require the loader to implement as part of any API.
After that what is needed varies based on the formats the module is available in. If there is bytecode you need to get said bytecode. If there is both bytecode and source you need the modification time of the source to see if the bytecode is stale. If there is only source or the bytecode is stale you need to be able to get the source itself. Finally, if the bytecode turned out to be bad (stale because its out of date or bad bytecode cookie) you need to be able to write out the new bytecode. So that's four things that the loader needs an API for: read bytecode, read source, source modification time, and writing out new bytecode.
Now the question becomes how should this API look. Originally, as I mentioned above, there was an idea of opaque token objects that represented paths that were passed around within the handler code and given back to the loader. This was to make the loader stateless and to minimize any repeated lookups on the filesystem (or whatever store the loader used).
But then I realized the loader did not need to be stateless for any reason. Plus if someone who implemented a loader wanted to optimize they could easily cache information in the loader themselves. That meant the API really only needed the module's name. That does add the possible inconvenience of verifying that a request was reasonable for that specific module (e.g., requesting the bytecode for a module who has no bytecode yet), but that should be minimal. And the biggest perk of just requiring the module's name is that PEP 302's optional extensions for importers and loaders can be used (the get_source method).
So that is the direction I am going with for my API change to importlib. I am also breaking up the filesystem importer and loader I wrote and having extension modules and py/pyc modules have their own importers and loaders. That way the generalization does not need to be so severe on everything. Plus extension modules are just plain different from py/pyc files and trying to make them work with the same API seems extreme.
Subscribe to:
Posts (Atom)