JavaScript is the new hotness. The big new trend in language works seems to be trying to come up with a new, fast JavaScript VM. See TraceMonkey, SquirrelFish Extreme (formerly just SquirrelFish), and V8. Like Python, JavaScript is highly dynamic. Could Python somehow learn from all of this work going into JavaScript and improve CPython's performance? Let's look at what each JavaScript VM does and see if it could potentially be applied to Python.
TraceMonkey's big trick is trace trees. When you do a loop, you get back to the top through jumps. Do this enough times and that jump is considered hot and of some use. When that decision is made, a trace is made from the top of the loop to the jump. That entire trace is then inlined.
Python could potentially do this if we ever figure out a good way to do inlining (Python's rich calling semantics make me wonder how easy it would be to do inlining, but I still think it is doable). I think the first step in trying to pull something like this off would be to get simple inlining working for functions and methods and build on that.
V8 does something called hidden classes. Basically, when a class is created a hidden version is made that makes attribute access be a memory offset. As more attributes are added, more hidden classes are created so each new hidden class allows for one more attribute to be accessed. The potential problem for doing this for Python is that namespace lookups are structured around using dicts. Being able to overload __dict__ is part of the design of the language. So if anything was done like this then a special namespace dict would be needed that could translate dict lookups to the proper memory offset lookup.
In the first announcement for SquirrelFish, the speedup came from moving off of AST execution and over to a bytecode interpreter. About the biggest difference between SquirrelFish and Python is the use of registers instead of a stack. I have always been curious as to whether using a register VM might be of some benefit to Python. SquirrelFish Extreme basically adds what V8 does with hidden classes.
There are two things that seem to be common amongst the VMs. One is using research done for the Self programming language. While the language itself is prototype-based, some of the work could still be applied to Python. And second, all of the interpreters use JIT compilation to native code when possible.
So what are the chances any of this will happen? Honestly, I don't know. The big advantage JavaScript development has over Python development is that JavaScript has major corporations sponsoring the work. Python, on the other hand, only has Guido is the only person who is paid to spend 50% of his time, when he has the time because it comes in spurts, to work on Python. Unless a huge number of us on the development team decide to take on some major VM work, we get a PhD student to jump in on this (and no, I am not that student unfortunately), or someone gets paid full-time to work on CPython and decides to tackle the performance question, I don't see any radical change to CPython's VM occurring. Here is to hoping I someday get paid to work on it. =)
Showing posts with label python language. Show all posts
Showing posts with label python language. Show all posts
2008-08-12
Podcast interview with me about Python is up
About a month ago Jonathan Johnson interviewed me for his 'Hello, World' podcast. That interview is now up and online.
This was my first interview so I have no clue how well I did. We also had some issues with sound but Jonathan told me he cleaned it up the best he could and it turned out pretty well (I quickly listened to the first few words from me and it sound intelligible). I think if I ever do another recorded interview using SIP or some other VoIP setup through my laptop I am going to get a USB microphone; I suspect my new Jawbone headset was not the best microphone choice.
I have to admit I have not heard the interview myself; I suspect if I do I will be going over in my head what I wished I had said instead and I would rather not do that to myself. But hopefully it is somehow informative and entertaining to others.
This was my first interview so I have no clue how well I did. We also had some issues with sound but Jonathan told me he cleaned it up the best he could and it turned out pretty well (I quickly listened to the first few words from me and it sound intelligible). I think if I ever do another recorded interview using SIP or some other VoIP setup through my laptop I am going to get a USB microphone; I suspect my new Jawbone headset was not the best microphone choice.
I have to admit I have not heard the interview myself; I suspect if I do I will be going over in my head what I wished I had said instead and I would rather not do that to myself. But hopefully it is somehow informative and entertaining to others.
2008-06-18
What are the ways at getting/setting a cell variable?
I have an idea brewing in my head, but it hinges on closures and preventing access to cell variables. What are the different ways on can either read or write to a cell variable from outside the closure?
Reading is pretty straightforward (I will be using Python 3.0 names): ``fxn.__func__.__closure__[0].cell_contents`` gives you direct access to the value of a cell variable. As for writing, that takes more effort:
Can anyone think up of other ways to gain read or write access to cell variables?
Reading is pretty straightforward (I will be using Python 3.0 names): ``fxn.__func__.__closure__[0].cell_contents`` gives you direct access to the value of a cell variable. As for writing, that takes more effort:
def innocent():
x = 0
class Foo:
def get(self):
return x
return Foo
cls = innocent()
ins = cls()
print('x =', ins.get())
def evil(cls):
def outside():
x = 0
def set(self, value):
nonlocal x
x = value
return set
import types
func = types.FunctionType(outside().__code__, {},
name=cls.__name__ + '.set', closure=cls.get.__closure__)
cls.set = func
evil(cls)
print('Setting to 42')
ins.set(42)
print('x =', ins.get())
Can anyone think up of other ways to gain read or write access to cell variables?
2007-11-03
Idea for process concurrency
This idea came to me this morning in bed and I have not completely thought it through, so I have no idea how reasonable it is. But I wanted to get it written down in case I decide to think about it some more.
I have been randomly thinking about how to get concurrency in Python using processes. I am not about to try to remove the GIL as it makes my life as a Python core developer easier. I know forking is out thanks to Windows lacking the system call. But the subprocess module shows it is not hard to launch an external process.
What if there was a function call that took a module name (just like an import statement) and returned an object that represented a concurrent call on that module? The returned object would have read and write methods on it as well as a close method.
On the other end would be a Python interpreter, executed with a command-line flag saying it was to act as a slave process. It would be given a file descriptor(s) representing OS sockets to communicate with the master process. Everything sent over the socket would be pickled data.
In this slave process the module specified to run would have its __name__ sent to '__spawn__' or '__slave__' or something other than '__main__'. That way the same module could be used for either a master or slave process. In the built-in namespace there would also be an object just like in the master process that presented the connection between the two.
The call starting all of this should probably have an argument specifying if exceptions sent over the line should be raised immediately or just considered data. There should also be an atexit function registered to make sure that the processes are cleaned up properly.
Basically I want to have a command-line flag that takes in info stating where an interpreter should pick up some communication point between two processes on the same machine that works on UNIX and Windows. The spawned process has a special __name__ value to signify that it is a spawned process. The two processes communicate by sending pickled data back and forth. I guess this is kind of like a poor man's Erlang or fork().
I have been randomly thinking about how to get concurrency in Python using processes. I am not about to try to remove the GIL as it makes my life as a Python core developer easier. I know forking is out thanks to Windows lacking the system call. But the subprocess module shows it is not hard to launch an external process.
What if there was a function call that took a module name (just like an import statement) and returned an object that represented a concurrent call on that module? The returned object would have read and write methods on it as well as a close method.
On the other end would be a Python interpreter, executed with a command-line flag saying it was to act as a slave process. It would be given a file descriptor(s) representing OS sockets to communicate with the master process. Everything sent over the socket would be pickled data.
In this slave process the module specified to run would have its __name__ sent to '__spawn__' or '__slave__' or something other than '__main__'. That way the same module could be used for either a master or slave process. In the built-in namespace there would also be an object just like in the master process that presented the connection between the two.
The call starting all of this should probably have an argument specifying if exceptions sent over the line should be raised immediately or just considered data. There should also be an atexit function registered to make sure that the processes are cleaned up properly.
Basically I want to have a command-line flag that takes in info stating where an interpreter should pick up some communication point between two processes on the same machine that works on UNIX and Windows. The spawned process has a special __name__ value to signify that it is a spawned process. The two processes communicate by sending pickled data back and forth. I guess this is kind of like a poor man's Erlang or fork().
2007-09-15
What would happen if we removed slicing/indexing syntax support?
I just finished reading (at least the parts that interested me) the paper 'An Overview of the Scala Programming Language'. As with most language papers I write, I read it from the perspective of a Python developer looking for new ideas to steal.
While nothing really leapt out at me (Scala is big into types and most of its interesting features ties into that), one thing that I thought was interesting is how there is no special syntactic support for array access in Scala; you don't use ``spam[42]``, but ``spam(42)``. It's an interesting idea and it is portable to Python.
Imagine that instead of having slicing/indexing support in the grammar, things such as lists, tuples, and dicts had their current __getitem__ method bound to __call__ instead. While odd, it is not totally a ridiculous thing to consider. By removing that piece of syntactic reliance you could know pass methods and functions to places where duck typing only calls for indexing. But as it stands now you can't do that unless you tack on a __getitem__ method to the object or catch the TypeError and then try calling the object.
You could even take this further and ditch the list and tuple literals syntax. ``[1,2,3]`` could, if list's constructor came to have arbitrary positional arguments, to be ``list(1,2,3)`` (you would need to change calls like ``list("abc")`` to be more like ``list(*"abc")`` in order to allow for this kind of instantiation). This would allow one to override the built-in list type with whatever one wanted and be guaranteed that all lists were constructed using your type. This could extend to tuples as well (dicts would be a pain because of their pairing nature and if tuple/list syntax support was gone that would get unwieldy very quickly without a cons type). This would hurt performance, though, as the literal syntax allows for list creation opcodes. And list comprehensions would just be replaced by generator expressions passed to the list type.
While I don't think getting rid of literal syntax for the buit-in types is good, it might not totally suck to ditch index/slice syntax support for making the built-in types callable. But then this has no chance of happening until Python 4 and who the hell knows when that would happen.
While nothing really leapt out at me (Scala is big into types and most of its interesting features ties into that), one thing that I thought was interesting is how there is no special syntactic support for array access in Scala; you don't use ``spam[42]``, but ``spam(42)``. It's an interesting idea and it is portable to Python.
Imagine that instead of having slicing/indexing support in the grammar, things such as lists, tuples, and dicts had their current __getitem__ method bound to __call__ instead. While odd, it is not totally a ridiculous thing to consider. By removing that piece of syntactic reliance you could know pass methods and functions to places where duck typing only calls for indexing. But as it stands now you can't do that unless you tack on a __getitem__ method to the object or catch the TypeError and then try calling the object.
You could even take this further and ditch the list and tuple literals syntax. ``[1,2,3]`` could, if list's constructor came to have arbitrary positional arguments, to be ``list(1,2,3)`` (you would need to change calls like ``list("abc")`` to be more like ``list(*"abc")`` in order to allow for this kind of instantiation). This would allow one to override the built-in list type with whatever one wanted and be guaranteed that all lists were constructed using your type. This could extend to tuples as well (dicts would be a pain because of their pairing nature and if tuple/list syntax support was gone that would get unwieldy very quickly without a cons type). This would hurt performance, though, as the literal syntax allows for list creation opcodes. And list comprehensions would just be replaced by generator expressions passed to the list type.
While I don't think getting rid of literal syntax for the buit-in types is good, it might not totally suck to ditch index/slice syntax support for making the built-in types callable. But then this has no chance of happening until Python 4 and who the hell knows when that would happen.
2007-08-31
Python 3.0a1 is out!
Guido pushed out a source distribution of Python 3.0a1 this morning! You can read the web page about the details, but basically this is a real alpha release. =) So please give it a spin and let us know if you find bugs.
2007-08-25
A little rant on __path__
What is the __path__ attribute on modules used for? One use is to identify a package from a module. Another is as an import optimization as __path__ replaces sys.path when importing within a package. Lastly, because of its position as a sys.path replacement, it allows for packages to tweak their search path.
Now why does this third use of __path__ slightly irk me? Well, while re-implementing zipimport I discovered it sets _path__ to ``zipfile_path/path_within_zip``. While in and of itself that is harmless, but when you have to deal with that as a pseudo entry on sys.path, it means you need to now figure out that is a package within a path. Obviously that path does not exactly work, so sys.path_hooks is consulted and (hopefully) a new importer can be found for that path.
But that is slightly annoying as you now have multiple importers for one zip file; one for the top-level zip file and then subsequent ones for every package contained within the zip file. This just seems like a waste. But then consider databases where the primary key is just the full module name. Having a unique __path__ value for every package is not really useful as all you really need is for __path__ to point back to the database, not to some unique path location as the DB can easily resolve where to find 'pkg.module' without having ``db_path/pkg`` for __path__.
Basically I am just thinking out loud about the idea of not relying on __path__ so much as representing some kind of directory path and more like a sys.path entry. This means that one should not necessarily expect modification of __path__ to work just as there is no guarantee that an entry on sys.path will find an importer from sys.path_hooks. This would allow for importers to exist that you only need one of but don't belong on sys.meta_path.
Now why does this third use of __path__ slightly irk me? Well, while re-implementing zipimport I discovered it sets _path__ to ``zipfile_path/path_within_zip``. While in and of itself that is harmless, but when you have to deal with that as a pseudo entry on sys.path, it means you need to now figure out that is a package within a path. Obviously that path does not exactly work, so sys.path_hooks is consulted and (hopefully) a new importer can be found for that path.
But that is slightly annoying as you now have multiple importers for one zip file; one for the top-level zip file and then subsequent ones for every package contained within the zip file. This just seems like a waste. But then consider databases where the primary key is just the full module name. Having a unique __path__ value for every package is not really useful as all you really need is for __path__ to point back to the database, not to some unique path location as the DB can easily resolve where to find 'pkg.module' without having ``db_path/pkg`` for __path__.
Basically I am just thinking out loud about the idea of not relying on __path__ so much as representing some kind of directory path and more like a sys.path entry. This means that one should not necessarily expect modification of __path__ to work just as there is no guarantee that an entry on sys.path will find an importer from sys.path_hooks. This would allow for importers to exist that you only need one of but don't belong on sys.meta_path.
2007-06-18
In case you ever wondered how pickle worked
Alexandre has a nice blog post up about how pickle works. I honestly had never bothered to learn enough about pickle to know why pickle existed over marshal since everything pickle has to represent has to be doable in bytecode. But pickle is definitely simpler.
Full disclosure: Alexandre is my student for the Google Summer of Code this year.
Full disclosure: Alexandre is my student for the Google Summer of Code this year.
2007-06-12
Import pseudocode now covers how modules are searched for
Previously I announced some pseudocode I tossed together to explain the core part of the import algorithm Python used. Turns out some people liked the post. So I decided to flesh it out the pseudocode some more.
In terms of the pre-existing pseudocode, I added support for how __path__ entries are handled. I also tried to clarify some comments.
This update includes code to explain how the file system is searched for an extension module or Python source or bytecode module. Probably the biggest thing to get out of this is how many stat calls Python has to make for each entry in sys.path. Two things come into play for this: file suffixes and packages.
Both extension modules and Python code have multiple file suffixes that can be valid. For extension modules, the typically valid suffixes are ".so" and "module.so" (on OS X; if I don't specify an OS assume I am talking about OS X or some other UNIX-based OS). That means that when import looks for the 'spam' module it has to look for 'spammodule.so' and 'spam.so' in the path on sys.path that is currently being considered. Same goes for Python code for the '.py' and either '.pyc' or '.pyo' depending. And you Windows users are even worse off as you have '.pyw' as well.
But what is being imported may be a package. That means you need to see if the module name is actually a directory name with an __init__ file (the current implementation only checks Python code files for this; importlib allows extension modules as well but that was for ease of coding reasons). That means you don't only check for 'spam.py' and 'spam.pyc', but also 'spam/__init__.py' and 'spam/__init__.pyc'.
All told that is a lot of stat calls per entry on sys.path. To cover extension modules you need two (one per suffix), and for Python code you need four (module or package check per source and bytecode suffix). That equals out to six stat calls per entry in sys.path in the worst case (it's actually worse in the C code as a stat for the directory for the package check adds one more, but importlib gets around it, I think, by just using os.path.isfile).
Luckily not all imports are top-level and thus hitting all of sys.path. If an import is happening within a package then you only need to check the __path__ attribute for the containing package (which is typically only a single directory).
Now some people blame Python's startup time on all of the imports the interpreter does to get itself going. And usually people blame the stat calls specifically. On NFS I can believe this. But as no one has bothered to implement a caching importer to cut out the caching and benchmark on a local hard disk to see if the stat calls are the problem.
My immediate worry with a caching importer is staleness. I want to be able to add a file to a directory on sys.path and have that file be noticed. I also want to be able to delete files and not have an importer think it is still there. I assume a stat call on the directory to check the modification time should pick up any changes to it and thus lead to a refreshing of the directory contents cache.
There is also the cost of getting the directory contents for the first time. I don't know how expensive that is compared to six stat calls. If it turns out to be more expensive in the common case it might not be worth it when there are not a ton of imports; there is a reason we have sys.modules.
Anyway, part of the reason I am really happy I wrote importlib is that it allows for easy experimentation like this. I am hoping to get to this at some point to see if it really does make a difference.
In terms of the pre-existing pseudocode, I added support for how __path__ entries are handled. I also tried to clarify some comments.
This update includes code to explain how the file system is searched for an extension module or Python source or bytecode module. Probably the biggest thing to get out of this is how many stat calls Python has to make for each entry in sys.path. Two things come into play for this: file suffixes and packages.
Both extension modules and Python code have multiple file suffixes that can be valid. For extension modules, the typically valid suffixes are ".so" and "module.so" (on OS X; if I don't specify an OS assume I am talking about OS X or some other UNIX-based OS). That means that when import looks for the 'spam' module it has to look for 'spammodule.so' and 'spam.so' in the path on sys.path that is currently being considered. Same goes for Python code for the '.py' and either '.pyc' or '.pyo' depending. And you Windows users are even worse off as you have '.pyw' as well.
But what is being imported may be a package. That means you need to see if the module name is actually a directory name with an __init__ file (the current implementation only checks Python code files for this; importlib allows extension modules as well but that was for ease of coding reasons). That means you don't only check for 'spam.py' and 'spam.pyc', but also 'spam/__init__.py' and 'spam/__init__.pyc'.
All told that is a lot of stat calls per entry on sys.path. To cover extension modules you need two (one per suffix), and for Python code you need four (module or package check per source and bytecode suffix). That equals out to six stat calls per entry in sys.path in the worst case (it's actually worse in the C code as a stat for the directory for the package check adds one more, but importlib gets around it, I think, by just using os.path.isfile).
Luckily not all imports are top-level and thus hitting all of sys.path. If an import is happening within a package then you only need to check the __path__ attribute for the containing package (which is typically only a single directory).
Now some people blame Python's startup time on all of the imports the interpreter does to get itself going. And usually people blame the stat calls specifically. On NFS I can believe this. But as no one has bothered to implement a caching importer to cut out the caching and benchmark on a local hard disk to see if the stat calls are the problem.
My immediate worry with a caching importer is staleness. I want to be able to add a file to a directory on sys.path and have that file be noticed. I also want to be able to delete files and not have an importer think it is still there. I assume a stat call on the directory to check the modification time should pick up any changes to it and thus lead to a refreshing of the directory contents cache.
There is also the cost of getting the directory contents for the first time. I don't know how expensive that is compared to six stat calls. If it turns out to be more expensive in the common case it might not be worth it when there are not a ton of imports; there is a reason we have sys.modules.
Anyway, part of the reason I am really happy I wrote importlib is that it allows for easy experimentation like this. I am hoping to get to this at some point to see if it really does make a difference.
2007-06-04
Now you too can know how import (roughly) works!
About two months ago I started trying to diagram import. The idea was twofold: I wanted something that people could look at so understanding how import worked overall was not a mystery, but I also wanted to show how complicated the thing is to help me argue for some changes I want to make.
To help me figure out what to diagram, I decided to write out the pseudocode to import. I figured I could write the pseudocode quickly to work out the level of detail I wanted for the diagram. That way I would waste less time experimenting in Graphviz which takes longer than experimenting in pseudocode.
The pseudocode turned out so well that I have now scrapped the diagram. I now have pseudocode that is syntactically proper Python with a ton of comments of what is going on. I then just make calls to functions that I have not defined that magically handle nitty-gritty details. This makes it easier to read than importlib as I can just cram it all into a single function without worrying about proper object abstractions, etc.
At this point I have written out the code for how import goes about looking for a module. This covers the use of sys.meta_path, sys.path, sys.path_importer_cache, and sys.path_hooks. I have not covered how the bytecode/source dance works or extension modules. Writing up those two will show why Python makes so many stat calls when it does an import.
To help me figure out what to diagram, I decided to write out the pseudocode to import. I figured I could write the pseudocode quickly to work out the level of detail I wanted for the diagram. That way I would waste less time experimenting in Graphviz which takes longer than experimenting in pseudocode.
The pseudocode turned out so well that I have now scrapped the diagram. I now have pseudocode that is syntactically proper Python with a ton of comments of what is going on. I then just make calls to functions that I have not defined that magically handle nitty-gritty details. This makes it easier to read than importlib as I can just cram it all into a single function without worrying about proper object abstractions, etc.
At this point I have written out the code for how import goes about looking for a module. This covers the use of sys.meta_path, sys.path, sys.path_importer_cache, and sys.path_hooks. I have not covered how the bytecode/source dance works or extension modules. Writing up those two will show why Python makes so many stat calls when it does an import.
2007-05-30
I have finished securing the Python interpreter!
Well, after many months of work I have finally managed to secure the Python interpreter as I laid out in my security paper! The code can be found in the bcannon-objcap branch in Python's svn repository (branched off the trunk).
Since I have finally reached this point I should give an overview of what I set out to do, what I have done, and how you too can secure the Python interpreter so that tangible resources (supposedly) can't get manipulated. If you want more thorough details about how the security mechanism is supposed to work, read the paper. This post is more about the technical details and to see if anyone can find a security hole.
What I Set Out To Do
The original goal of this work was to come up with a way so that you could run Python code in an embedded Python interpreter and not worry about it opening arbitrary sockets or touching any files unless you explicitly allowed it. The hope was to get this working for applications that embed the Python interpreter so that it could be used as a springboard into allowing people to have a custom interpreter to run Python apps in securely.
This work was never meant to be an rexec replacement. While I always kept rexec in the back of my mind to make sure that something I did would not explicitly prevent some rexec solution from coming forward, it was never the end goal.
Nor was it a goal to protect intangible things such as memory or CPU usage. This was supposed to protect stuff like files and sockets; things with a concrete object representation.
Changes to Python
So, what the heck did I end up doing? In terms of Python itself, it was actually very minor. First I removed the constructor for the file type. I never thought that I could hide the file type properly, so I just crippled it so that you had to go through open() to get a initialised file object. I added a module called objcap that includes an initialisation function for allocated file objects if people really feel the need to not use open().
Second I removed the constructor from code. I added a function to objcap to deal with this. I did this as Python does not verify bytecode and so someone might be able to crash the interpreter or something with some crazy bytecode.
With those two types neutered, I turned my attention to protecting imports. With my Python implementation of import, importlib, I already knew I had the control I needed to prevent dangerous imports, but I had to make sure that the fully powered import was not exposed in the interpreter or that attributes were exposed. I wrote a simple delegate in C that I stored at sys.import_delegate that simply called what was stored at sys.import_ . This way no attributes on the callable object were exposed. Putting all in the sys module was in no way required, but it was the simplest solution for me. It could have all easily been implemented externally of the sys module if I felt like putting in the effort. =)
I also had to edit codecs so that it didn't import sys but just the one attribute it needed. That way if you imported codecs you didn't get access to the sys module for free.
The last change to Python itself was how sys was re-imported. With sys being so special it has its module dict stored with the interpreter instance. Also because sys is special there are several places in the codebase that add stuff to the sys module during interpreter initialisation. The problem is that the built-in import machinery (as it is exposed through the imp module and thus affecting importlib) caches built-in modules' dicts and some stuff gets added to sys' dict after the caching. That means if you delete the sys module and re-import it can be in bad shape. So I had to special case re-importing the sys module. This breaks calling reload() on sys, but test_xmlrpc is the only thing that I know that does that and reload() is going away in Python 3.0, so I don't care. =)
And that's it for the changes required within Python itself. It really is not extensive in any way. I also don't see a huge issue with getting the key parts (the file and code changes) into the core as long as what is needed is exposed in a reasonable extension module.
Tweaking the Interpreter
Where most of the work comes in is in tweaking interpreter stuff outside of the core code. If you look at secure_python.c in the bcannon-objcap branch you can see what is required to get a secure version of Python to run in an embedded C application with the above-mentioned changes.
Obviously the first step is to initialize the interpreter. That's easy.
Next step is to set importlib as the import machinery. That takes creating a whitelist of built-in, frozen, and extension modules you want to allow (6, 0, and 19 each, respectively, that I could find would be safe), setting an instance of controlled_importlib.ControlledImport to sys.import_, and setting __import__ to sys.import_delegate. Now all imports go through importlib which makes sure that imports are controlled. I also clear sys.meta_path and sys.path_hooks to make sure no lingering imports are there that would accidentally subvert the whitelisting.
Next sys.modules needs to be cleaned out. Starting up Python leads to a bunch of modules being imported. Most are not critical once the interpreter is up. But a handful are required for Python to work. Those required modules (__builtin__, __main__, encodings, codecs, _codecs) are left in sys.modules. The rest are swept into a dict stored in sys.modules under the ".hidden" key. That keeps the objects alive without letting them be imported. The warnings module also gets imported and then moved as it needs to get cached at the C level.
With that done sys.path_importer_cache gets cleared. I leave sys.path alone so that I can import stuff from the stdlib without issue, but it can obviously be tweaked.
Finally, open(), execfile(), and SystemExit are removed from the built-in namespace. The first two are because they open files indiscriminately. SystemExit goes away because the Python interpreter automatically tears down the interpreter if it propagates all the way up. And by not whitelisting the exceptions module you shouldn't be able to get SystemExit.
And that's that. As you can see a lot of it is externally done to Python thanks to how import statements actually call __import__ and how Python exposes so much as dictionaries.
Building and Testing
As for confirming all of this works, I have some tests in the branch. To build all of this you can run build_secure_py.sh, but this has only been used on OS X. And to run the tests execute run_security_tests.py with a *regular* Python interpreter. Stuff in tests/succeed are expected to work while stuff in tests/fail require the code being tested to be in a try statement.
Wrap-Up
If anyone checks out the code, runs it, and manages to find a way to open a file, socket, create a code object from scratch, or arbitrarily import any extension, frozen, or built-in module, please let me know! Hopefully nobody finds a way and this all holds up. If it does I will begin trying to get what I need into the core so that this work doesn't require a special checkout of Python.
Since I have finally reached this point I should give an overview of what I set out to do, what I have done, and how you too can secure the Python interpreter so that tangible resources (supposedly) can't get manipulated. If you want more thorough details about how the security mechanism is supposed to work, read the paper. This post is more about the technical details and to see if anyone can find a security hole.
What I Set Out To Do
The original goal of this work was to come up with a way so that you could run Python code in an embedded Python interpreter and not worry about it opening arbitrary sockets or touching any files unless you explicitly allowed it. The hope was to get this working for applications that embed the Python interpreter so that it could be used as a springboard into allowing people to have a custom interpreter to run Python apps in securely.
This work was never meant to be an rexec replacement. While I always kept rexec in the back of my mind to make sure that something I did would not explicitly prevent some rexec solution from coming forward, it was never the end goal.
Nor was it a goal to protect intangible things such as memory or CPU usage. This was supposed to protect stuff like files and sockets; things with a concrete object representation.
Changes to Python
So, what the heck did I end up doing? In terms of Python itself, it was actually very minor. First I removed the constructor for the file type. I never thought that I could hide the file type properly, so I just crippled it so that you had to go through open() to get a initialised file object. I added a module called objcap that includes an initialisation function for allocated file objects if people really feel the need to not use open().
Second I removed the constructor from code. I added a function to objcap to deal with this. I did this as Python does not verify bytecode and so someone might be able to crash the interpreter or something with some crazy bytecode.
With those two types neutered, I turned my attention to protecting imports. With my Python implementation of import, importlib, I already knew I had the control I needed to prevent dangerous imports, but I had to make sure that the fully powered import was not exposed in the interpreter or that attributes were exposed. I wrote a simple delegate in C that I stored at sys.import_delegate that simply called what was stored at sys.import_ . This way no attributes on the callable object were exposed. Putting all in the sys module was in no way required, but it was the simplest solution for me. It could have all easily been implemented externally of the sys module if I felt like putting in the effort. =)
I also had to edit codecs so that it didn't import sys but just the one attribute it needed. That way if you imported codecs you didn't get access to the sys module for free.
The last change to Python itself was how sys was re-imported. With sys being so special it has its module dict stored with the interpreter instance. Also because sys is special there are several places in the codebase that add stuff to the sys module during interpreter initialisation. The problem is that the built-in import machinery (as it is exposed through the imp module and thus affecting importlib) caches built-in modules' dicts and some stuff gets added to sys' dict after the caching. That means if you delete the sys module and re-import it can be in bad shape. So I had to special case re-importing the sys module. This breaks calling reload() on sys, but test_xmlrpc is the only thing that I know that does that and reload() is going away in Python 3.0, so I don't care. =)
And that's it for the changes required within Python itself. It really is not extensive in any way. I also don't see a huge issue with getting the key parts (the file and code changes) into the core as long as what is needed is exposed in a reasonable extension module.
Tweaking the Interpreter
Where most of the work comes in is in tweaking interpreter stuff outside of the core code. If you look at secure_python.c in the bcannon-objcap branch you can see what is required to get a secure version of Python to run in an embedded C application with the above-mentioned changes.
Obviously the first step is to initialize the interpreter. That's easy.
Next step is to set importlib as the import machinery. That takes creating a whitelist of built-in, frozen, and extension modules you want to allow (6, 0, and 19 each, respectively, that I could find would be safe), setting an instance of controlled_importlib.ControlledImport to sys.import_, and setting __import__ to sys.import_delegate. Now all imports go through importlib which makes sure that imports are controlled. I also clear sys.meta_path and sys.path_hooks to make sure no lingering imports are there that would accidentally subvert the whitelisting.
Next sys.modules needs to be cleaned out. Starting up Python leads to a bunch of modules being imported. Most are not critical once the interpreter is up. But a handful are required for Python to work. Those required modules (__builtin__, __main__, encodings, codecs, _codecs) are left in sys.modules. The rest are swept into a dict stored in sys.modules under the ".hidden" key. That keeps the objects alive without letting them be imported. The warnings module also gets imported and then moved as it needs to get cached at the C level.
With that done sys.path_importer_cache gets cleared. I leave sys.path alone so that I can import stuff from the stdlib without issue, but it can obviously be tweaked.
Finally, open(), execfile(), and SystemExit are removed from the built-in namespace. The first two are because they open files indiscriminately. SystemExit goes away because the Python interpreter automatically tears down the interpreter if it propagates all the way up. And by not whitelisting the exceptions module you shouldn't be able to get SystemExit.
And that's that. As you can see a lot of it is externally done to Python thanks to how import statements actually call __import__ and how Python exposes so much as dictionaries.
Building and Testing
As for confirming all of this works, I have some tests in the branch. To build all of this you can run build_secure_py.sh, but this has only been used on OS X. And to run the tests execute run_security_tests.py with a *regular* Python interpreter. Stuff in tests/succeed are expected to work while stuff in tests/fail require the code being tested to be in a try statement.
Wrap-Up
If anyone checks out the code, runs it, and manages to find a way to open a file, socket, create a code object from scratch, or arbitrarily import any extension, frozen, or built-in module, please let me know! Hopefully nobody finds a way and this all holds up. If it does I will begin trying to get what I need into the core so that this work doesn't require a special checkout of Python.
2007-05-19
Bloody PythonLauncher screwed Terminal.app!
I was swearing my head off, scaring my girlfriend while I was at since I usually don't go off the handle that badly, when PythonLauncher screwed Terminal.app. For the non-OS X users out there, PythonLauncher is an application that comes with Python on OS X that lets you double-click a Python script to launch it. Terminal.app is the terminal application in OS X. Since I do most of my coding using a terminal not having it work was VERY frustrating.
For some unknown reason Terminal.app was running the command that PythonLauncher used to execute Python scripts. That was especially painful for me as I use keychain and thus need to enter my password for my SSH 2 key when my shell launches if I do not have an ssh-agent running yet. Having some command get pasted in instead and lock up the shell did not make me happy.
If anyone ever has troubles like this (closest I found was someone sending an email about how pythonlauncher possessed terminal), the solution is to use the Property List Editor that comes with the Apple Developer Tools (/Developer/Applications/Utilities/Property List Editor) to edit the plist for Terminal (~/Library/Preferences/com.apple.Terminal.plist). If you look you will find an executeString key with a value of what PythonLauncher uses. Delete that key, save the plist, and Terminal.app should then be fixed. Probably best to edit the file with Terminal.app closed.
For some unknown reason Terminal.app was running the command that PythonLauncher used to execute Python scripts. That was especially painful for me as I use keychain and thus need to enter my password for my SSH 2 key when my shell launches if I do not have an ssh-agent running yet. Having some command get pasted in instead and lock up the shell did not make me happy.
If anyone ever has troubles like this (closest I found was someone sending an email about how pythonlauncher possessed terminal), the solution is to use the Property List Editor that comes with the Apple Developer Tools (/Developer/Applications/Utilities/Property List Editor) to edit the plist for Terminal (~/Library/Preferences/com.apple.Terminal.plist). If you look you will find an executeString key with a value of what PythonLauncher uses. Delete that key, save the plist, and Terminal.app should then be fixed. Probably best to edit the file with Terminal.app closed.
2007-05-02
__import_ex__
I thought some more today about how I would replace __import__ and tweak the IMPORT_NAME and IMPORT_FROM bytecodes. I am not going to be writing a PEP for any of this for a while, but I figured I might as well write out what I am thinking in case anyone else is interested in this topic.
First, the signature for __import__ needs a major overhaul. Since tweaking the signature is out of the question because of backwards-compatibility I figured I could introduce a new function, __import_ex__(name:Sequence[str], caller__name__:str, caller__path__:(Sequence[str]|None)=None) -> object. Here is an explanation of the parameters:
But it's different for ``from spam.bacon import python``. That uses two bytecodes: IMPORT_NAME and IMPORT_FROM. The first bytecode gets 'spam.bacon' on to the stack. The second gets 'python' off of 'spam.bacon' and puts that on the stack to be bound to the name 'python'.
As you may have noticed, IMPORT_NAME puts different things on the stack based on what type of import statement is called. To me that is kind of nasty. Why can't you just return the root module of what is imported? Well, think of a relative import like '...blah' that resolves to 'foo.bar.blah'. If you just put on the root module then you have 'foo' on the stack. But there is no way to introspect that you are going to what stuff of of the 'foo.bar.blah' module.
What to do? Well, __import_ex__ could return a tuple of the module and the absolute name of what was requested, but that seems wasteful. One could add a flag to __import_ex__ to signify that leaf module is to be returned instead of the root, much like how the presense of fromlist in __import__ does now. That would let it work much like it does now.
Or __import_ex__ could always return the leaf module. Then a new bytecode could be introduced to return the root module for a leaf module that was introduced. This is as easy as ``sys.modules[mod.__name__.partition('.')[0]]``.
Either of the last two options work. Either a flag goes on to __import_ex__ to flag exactly what module is to be returned, or we end up with more fine-grained bytecode. I personally vote for the latter. You would end up with bytecode like this:
First, the signature for __import__ needs a major overhaul. Since tweaking the signature is out of the question because of backwards-compatibility I figured I could introduce a new function, __import_ex__(name:Sequence[str], caller__name__:str, caller__path__:(Sequence[str]|None)=None) -> object. Here is an explanation of the parameters:
- name : A list of strings representing the parts of the module to import. Empty strings at the front of the list represent a dot in the name. It is a list of strings instead of just a string so as to have the bytecode handle the string splitting, making the function have one less thing to deal with. Plus if you are programmatically creating an import there is just as much of a chance (if not more) that you are building up the import and thus probably would prefer to work with a list than do a bunch of string manipulations. But the parameter can easily be changed to be a string.
- caller__name__ : The name of the module requesting the import.
- caller__path__ : The value of __path__ if the caller has it defined, else None. This could be a boolean instead of the actual value stored in __path__. The reason for not doing that is that import will end up needing the value from __path__ anyway, so might as well just grab it now instead of having to fetch the caller from sys.modules and getting the path value then. Plus it helps facilitate testing by having less things happen behind the scenes.
But it's different for ``from spam.bacon import python``. That uses two bytecodes: IMPORT_NAME and IMPORT_FROM. The first bytecode gets 'spam.bacon' on to the stack. The second gets 'python' off of 'spam.bacon' and puts that on the stack to be bound to the name 'python'.
As you may have noticed, IMPORT_NAME puts different things on the stack based on what type of import statement is called. To me that is kind of nasty. Why can't you just return the root module of what is imported? Well, think of a relative import like '...blah' that resolves to 'foo.bar.blah'. If you just put on the root module then you have 'foo' on the stack. But there is no way to introspect that you are going to what stuff of of the 'foo.bar.blah' module.
What to do? Well, __import_ex__ could return a tuple of the module and the absolute name of what was requested, but that seems wasteful. One could add a flag to __import_ex__ to signify that leaf module is to be returned instead of the root, much like how the presense of fromlist in __import__ does now. That would let it work much like it does now.
Or __import_ex__ could always return the leaf module. Then a new bytecode could be introduced to return the root module for a leaf module that was introduced. This is as easy as ``sys.modules[mod.__name__.partition('.')[0]]``.
Either of the last two options work. Either a flag goes on to __import_ex__ to flag exactly what module is to be returned, or we end up with more fine-grained bytecode. I personally vote for the latter. You would end up with bytecode like this:
- IMPORT_NAME : Pop the name of the module to import off the stack and push that module.
- IMPORT_ROOT : Pop a module off of the stack and push the root module.
- IMPORT_FROM : Pop a module and an list of items to get from that module and push those items requested. If the specified item is not on the object, try to do an import as needed to get it.
2007-04-23
Five (+ 1) Things I Hate About Python (and then five more)
I was poking around Jacob Kaplan-Moss' site and read an entry he did on five thing he hates about Python (which was inspired by Titus Brown who was inspired by Brian D. Foy's post about how every language advocate should know what they don't like about their favourite language). I thought the idea was sound and as I am a Python bigot I figured it would only be fair if I publicly stated five things about the Python language that I hate.
Overlap between Jacob, Titus, and me is interesting. Granted I wrote my list shortly after reading both of theirs, but I doubt my list would change that much. We all didn't like how the stdlib is documented. I would have complained about multi-processing, but since I don't do it that often and I get the perks of the GIL making my life easier I don't find it a flaw in the language.
But what is more interesting is what could be fixed. Pretty much short of the multi-processing/GIL complaint no one is hating on anything that couldn't be fixed with some determination. Trick is finding someone with that determination. =)
And as a bonus, I have the five things I hate about Python that make my life as a core developer a pain but I know won't ever change:
- stdlib documentation. The docs for the stdlib are incomplete, period. They tend to also be underspecified in terms of semantics. It sucks that I sometimes have to check source code to see exactly what a function or method will do in a certain circumstance.
- Organization of the stdlib. I really don't like how friggin' flat the stdlib's namespace is. I have tried twice now to get it into a nested (but shallow) namespace. But every time you run into the flat camp and the deeply nested camp on top of people bickering over how to separate modules. I am personally burned out on this and won't move forward unless I either get huge backing from python-dev or Guido to give guidance over what he is after because I am not in the mood to fight for this just to have everyone throw out their opinion but not be willing to compromise on anything.
- C API inconsistency. When does a C function return a borrowed reference? When does it not? Should the caller or callee verify arguments are correct? When can a function not have an error return value? What are the rules for -1 being an error compared to 0 or NULL? We need some rules and we need to stick to them! If I know the name of the function I want to use I shouldn't have to look at the docs for stuff like this.
- Complexity of method call resolution. Multiple inheritance, __slots__, descriptors (both data and non-data), super, metaclasses. Python has a rich set of ways for you to muck with what gets called (both at call time and at class creation time). While everything listed has a purpose and is useful, trying to keep it all straight can be hard. And trying to maintain the code that implements all of this is *really* hard.
- How outdated the stdlib is. Names don't follow the style guide. Stuff that has not been touched in years is sitting in there. New ways of doing things have come about. Why the hell do we need all of it? I want a stdlib that is slim, up-to-date, and can move cruft out when a better solution comes along.
- BONUS: Plethora of C functions/macros just for performance reasons. Do you really need macro versions of functions that access an item in a tuple? Or how about a bazillion ways to create a string? The C API needs to go on a diet.
Overlap between Jacob, Titus, and me is interesting. Granted I wrote my list shortly after reading both of theirs, but I doubt my list would change that much. We all didn't like how the stdlib is documented. I would have complained about multi-processing, but since I don't do it that often and I get the perks of the GIL making my life easier I don't find it a flaw in the language.
But what is more interesting is what could be fixed. Pretty much short of the multi-processing/GIL complaint no one is hating on anything that couldn't be fixed with some determination. Trick is finding someone with that determination. =)
And as a bonus, I have the five things I hate about Python that make my life as a core developer a pain but I know won't ever change:
- Refcounting. Some people love the fact that Python cleans up after itself so quickly. That's nice. But getting refcounts right takes a lot of practice. And tracking a reference leak down can be truly painful and requires its own skill set. If we could compartmentalize memory issues more and use something more transparent I would be rather happy as I wouldn't have to deal with memory anymore.
- Use of C. The way we have to fake objects, its lax typing, its crappy library, and huge amount of undefined behaviour makes using C a pain. I personally would rather be using something like C++ (which I hate as a language but realize it can be useful when you code to a *small* subset) or Objective-C where basic object stuff is done for us and stuff like memory management can be made easier (refcounting in C++ is so much easier thanks to RAII). Hell, I would be okay with certain functional languages (Haskell is *not* on that list). But I would not mind having the implementation language do more for us (and no, I am not moving over to PyPy as I like clear delineations between implementation and target languages, which involves a long story as to why I feel that way, and I am willing to work at a low level for overall performance).
- Backwards compatibility. Users love, I hate it. Why? It makes my life hard. I have to worry about whether a change I make will make someone's code no longer work. That can prevent me from doing the better solution. It also leads to outdated APIs that then must be supported along with the more modern one that is better.
- Extensive platform support. I don't care about HP-UX, AIX, or any other random flavour of UNIX. I hate all of the OS-specific macros peppered throughout Python. It is a pain to try to trace code through it and it makes the source a mess.
- The build system. I hate autoconf. I hate makefiles. Our own setup.py is a bloody mess. I don't know how to get off of autoconf to something that runs on almost any platform. I don't even want to attempt to untangle our makefile to move over to cmake or bakefiles or whatever. I am just glad I have never severely broken the build system before.
2007-04-19
Python security paper online
This past term I audited a grad course on computer security here at UBC in the EECE dept. As part of the course there was a final paper. Having already spent months on the topic of security and Python I used the course as an excuse to write up my work.
I have put the paper, "Controlling Access to Resources Within The Python Interpreter" online. There are some things you should keep in mind when reading the paper. One is that this paper was for a course, not a journal or conference. Another is that I had an eight page limit so I didn't have space to go into how the security implementation would defend against common attacks, etc. Lastly, the audience did not know Python, so there is some stuff in the paper that is probably rather basic for anyone who reads this blog.
I think the paper turned out fine. Comments on the security design are welcome. Edit issues are not too critical as this paper would most likely get reworked for a conference if it ever comes to this.
Assuming I have the time I am hoping to use this paper as a reference in a PEP to get the changes I want into Py3K. But I am not sure if I am going to have the time to pull this off by April 30th, especially if I want a proof-of-concept ready by then. So if it slips to Python 3.1/2.7 then that is life.
And a special thanks needs to go to my supervisor, Eric Wohlstadter, for funding me while doing this work. Even when it seemed we might not get a publication out of this he allowed me to continue to work on it which I really appreciate.
I have put the paper, "Controlling Access to Resources Within The Python Interpreter" online. There are some things you should keep in mind when reading the paper. One is that this paper was for a course, not a journal or conference. Another is that I had an eight page limit so I didn't have space to go into how the security implementation would defend against common attacks, etc. Lastly, the audience did not know Python, so there is some stuff in the paper that is probably rather basic for anyone who reads this blog.
I think the paper turned out fine. Comments on the security design are welcome. Edit issues are not too critical as this paper would most likely get reworked for a conference if it ever comes to this.
Assuming I have the time I am hoping to use this paper as a reference in a PEP to get the changes I want into Py3K. But I am not sure if I am going to have the time to pull this off by April 30th, especially if I want a proof-of-concept ready by then. So if it slips to Python 3.1/2.7 then that is life.
And a special thanks needs to go to my supervisor, Eric Wohlstadter, for funding me while doing this work. Even when it seemed we might not get a publication out of this he allowed me to continue to work on it which I really appreciate.
2007-03-21
A possible change to handle "main" code in a module
I think every Python programmer learns early on the ``if __name__ == '__main__'`` idiom. It's fairly simple and it makes sense when you explain what is happening. Plus it nicely ties into the fact that Python modules are actually executed when imported.
The problem is that they partially break down when using the new relative imports (e.g., ``from .. import foo``). The problem is that to calculate the absolute name of the foo module being imported, import takes the caller's __name__ value from its global namespace, splits on '.', and then subtracts off parts on have many levels up the import needs to go to search for the module.
But when the module with a relative import has it's name set as '__main__', the name resolution for relative imports doesn't work (you end up with an ImportError). This sucks when you want to use a file in a package as an executable script for something. How can this be solved?
The idea I came up with while this was discussed on python-ideas (don't remember how long ago; possibly two months?) was to make __main__ a built-in variable that stored the name of the module currently being executed. This would allow the idiom to be changed to ``if __name__ == __main__``. This has a perk of no longer having to set the '__main__' key in sys.modules any longer. It also prevents errors by mistyping the string '__main__' and instead relying on a variable name.
Obviously another option is to go with defining a special function name that automatically gets executed when the module is first imported. I personally, though, don't like that solution as much.
Anyway, if feedback on this post is positive I will propose this back on python-ideas and then hopefully to python-3000 after that.
The problem is that they partially break down when using the new relative imports (e.g., ``from .. import foo``). The problem is that to calculate the absolute name of the foo module being imported, import takes the caller's __name__ value from its global namespace, splits on '.', and then subtracts off parts on have many levels up the import needs to go to search for the module.
But when the module with a relative import has it's name set as '__main__', the name resolution for relative imports doesn't work (you end up with an ImportError). This sucks when you want to use a file in a package as an executable script for something. How can this be solved?
The idea I came up with while this was discussed on python-ideas (don't remember how long ago; possibly two months?) was to make __main__ a built-in variable that stored the name of the module currently being executed. This would allow the idiom to be changed to ``if __name__ == __main__``. This has a perk of no longer having to set the '__main__' key in sys.modules any longer. It also prevents errors by mistyping the string '__main__' and instead relying on a variable name.
Obviously another option is to go with defining a special function name that automatically gets executed when the module is first imported. I personally, though, don't like that solution as much.
Anyway, if feedback on this post is positive I will propose this back on python-ideas and then hopefully to python-3000 after that.
Informal poll: interface of exceptions
I made three mistakes when removing indexing/slicing on exceptions. When I came across ``exc[0]`` I translated it to ``exc.message`` instead of ``exc.args[0]``. There is an important difference, though, between using 'message' and 'args' as the former can be the empty string while args[0] has a value if more than a single argument is passed in. I had not even realized I had made the mistake until Guido pointed it out to me.
With this subtlety, along with how painful it will be to remove 'args' from exceptions (which I have done for the core and anyone at the PyCon sprint can tell you frustrated me to no end), my question is would people miss 'message'? As of right now it has odd semantics because it was added in 2.5 in hopes of removing 'args' and having exceptions only accept a single argument by default, thus we didn't want people to rely on 'message' and think that only one argument was passed in. But if 'args' ends up staying because the transition off of it cannot be done reasonably then is 'message' even worth keeping? The attribute could change to just args[0], but that does not seem useful.
If you happen to have an opinion please leave a comment.
With this subtlety, along with how painful it will be to remove 'args' from exceptions (which I have done for the core and anyone at the PyCon sprint can tell you frustrated me to no end), my question is would people miss 'message'? As of right now it has odd semantics because it was added in 2.5 in hopes of removing 'args' and having exceptions only accept a single argument by default, thus we didn't want people to rely on 'message' and think that only one argument was passed in. But if 'args' ends up staying because the transition off of it cannot be done reasonably then is 'message' even worth keeping? The attribute could change to just args[0], but that does not seem useful.
If you happen to have an opinion please leave a comment.
2007-03-03
PEP 3113 (removal of automatic tuple parameter unpacking)
Anyone at PyCon who saw my lightning talk or was around me throughout the conference will know what this PEP is about: getting rid of tuple parameters. Thanks to everyone at PyCon who provided arguments for their removal (basically Ka-Ping Yee was the only person who provided any argument for them to stay).
Collin Winter has already implemented a transform rule for 2to3 and Guido has already said he supports the idea. It looks like this PEP will get accepted for Py3K! Woohoo!
Collin Winter has already implemented a transform rule for 2to3 and Guido has already said he supports the idea. It looks like this PEP will get accepted for Py3K! Woohoo!
2007-02-16
I really want to change the signature of __import__
One of the things that bugged me when I was rewriting import was the __import__ function's signature: ``__import__(module, globals, locals, fromlist, level)``. To me, that signature is overly generalized beyond what is reasonable. Three of those five arguments could be changed, possibly for the better.
[Please note, that when I mention implementation-specific stuff below, I am thinking about my Python implementation and not the C version. There is a chance there is a slight deviation between the two.]
First, the module. Right now it is a string of the module being requested. But a possibly handy change would be to make it a sequence of the parts of a path. Instead of passing in "pkg.mod", why not ['pkg', 'mod']? It makes the parts of the path explicit and cuts down on possible errors doing string manipulation with a dot in order to make the full path that you want.
Second, the caller's globals namespace is not entirely needed. The only thing that is looked at is __path__ and __name__. __path__ is checked for so as to know if one is importing from within a package's __init__ file. __name__ is used to resolve relative imports to an absolute name. All other possible values you get from the global namespace of the valler is ignored.
And even the __path__ check is not truly needed; it's used in an optimized fashion. If you considered sys.modules to contain the authoritative versions of modules, you really only need __name__. With that you can check sys.modules for the module doing the call and get __path__ from there.
If you really wanted the globals you could still easily get it from __name__ anyway. Just grab the module from sys.modules and see what it contains.
The locals passed in are entirely ignored. If you really wanted them, though, you could get the execution frame and go backwards to the caller and look at the locals that way.
The fromlist is actually fine. The only odd thing about it is when '*' is used. But otherwise it is okay.
For level, that works as well. But, if the sequence version of the module name were used, even this would not be needed. The empty string could represent a dot and thus you could ditch level and base the relative import on the number of empty strings at the beginning of the sequence object. This is, coincidentally, the semantics that str.join has when splitting on a string that happens to be repeated.
In other words, the signature could be simplified to ``__import__(module_path, caller_name, from_list)``. I personally think that is a whole lot cleaner and easier to work with.
The problem with even considering this change is that I don't know if people who have written wrappers to __import__ use globals or locals. If they do, and they are used by most people, then that kills this idea. But if not many people do, then a new __import__ function can be introduced with a new name, and the pre-existing __import__ can be changed to use the new function.
Does anyone out there know of custom __import__ wrappers that use globals or locals for something? Or do all of the __import__ functions just not do anything special?
[Please note, that when I mention implementation-specific stuff below, I am thinking about my Python implementation and not the C version. There is a chance there is a slight deviation between the two.]
First, the module. Right now it is a string of the module being requested. But a possibly handy change would be to make it a sequence of the parts of a path. Instead of passing in "pkg.mod", why not ['pkg', 'mod']? It makes the parts of the path explicit and cuts down on possible errors doing string manipulation with a dot in order to make the full path that you want.
Second, the caller's globals namespace is not entirely needed. The only thing that is looked at is __path__ and __name__. __path__ is checked for so as to know if one is importing from within a package's __init__ file. __name__ is used to resolve relative imports to an absolute name. All other possible values you get from the global namespace of the valler is ignored.
And even the __path__ check is not truly needed; it's used in an optimized fashion. If you considered sys.modules to contain the authoritative versions of modules, you really only need __name__. With that you can check sys.modules for the module doing the call and get __path__ from there.
If you really wanted the globals you could still easily get it from __name__ anyway. Just grab the module from sys.modules and see what it contains.
The locals passed in are entirely ignored. If you really wanted them, though, you could get the execution frame and go backwards to the caller and look at the locals that way.
The fromlist is actually fine. The only odd thing about it is when '*' is used. But otherwise it is okay.
For level, that works as well. But, if the sequence version of the module name were used, even this would not be needed. The empty string could represent a dot and thus you could ditch level and base the relative import on the number of empty strings at the beginning of the sequence object. This is, coincidentally, the semantics that str.join has when splitting on a string that happens to be repeated.
In other words, the signature could be simplified to ``__import__(module_path, caller_name, from_list)``. I personally think that is a whole lot cleaner and easier to work with.
The problem with even considering this change is that I don't know if people who have written wrappers to __import__ use globals or locals. If they do, and they are used by most people, then that kills this idea. But if not many people do, then a new __import__ function can be introduced with a new name, and the pre-existing __import__ can be changed to use the new function.
Does anyone out there know of custom __import__ wrappers that use globals or locals for something? Or do all of the __import__ functions just not do anything special?
Subscribe to:
Posts (Atom)