2006-01-18

Adding simple concurrency to Python

So, how could one add simple concurrency to Python? Well, decorators make turning a method or function into one that executes in a thread rather straight-forward. You then just return a future and all is mostly well (the only issue is type-checking, but a metaclass might be possible to solve that issue by generating a future class with the proper inheritance tree, if that is even needed).

But what about global variables? Those pesky things that always get in the way of simple, concurrent code? Well, decorators help with this. Since they are passed the method or function, you can view the bytecode. That means you can look up global variable accesses. You can also get the names of the globals easily enough. So, if you then wrap those globals in a locking object then you could let the concurrent method have exclusive access to the global and not worry about the global changing its value underneath the concurrent method call. And as part of the cleanup of the concurrent method, you unlock the global, and thus allow execution in the main thread to continue.

Now this is not fool-proof. Someone storing a global somewhere will be able to circumvent the protection unless __getattribute__() is injected into the global properly and hooks into the proper lock (which might be tricky in situations where it is a C type or an object that uses slots and thus cannot have an aribtrary attribute added to hold the lock). Could have a module that stores all of this and also acts as a global lock holder. That way a generic __getattribute__() can be injected and that can then look up in the module for the lock needed by the global. But then again if it is read-only it still does not solve the problem. Bah!

It might also be easier to specify the globals as part of the decorator. That would allow for explicit locking of the globals with a locking equivalent object. Might also simplify the whole situation.

But for the base case, it could make it fairly straight-forward to add simple concurrency.