This idea came to me this morning in bed and I have not completely thought it through, so I have no idea how reasonable it is. But I wanted to get it written down in case I decide to think about it some more.
I have been randomly thinking about how to get concurrency in Python using processes. I am not about to try to remove the GIL as it makes my life as a Python core developer easier. I know forking is out thanks to Windows lacking the system call. But the subprocess module shows it is not hard to launch an external process.
What if there was a function call that took a module name (just like an import statement) and returned an object that represented a concurrent call on that module? The returned object would have read and write methods on it as well as a close method.
On the other end would be a Python interpreter, executed with a command-line flag saying it was to act as a slave process. It would be given a file descriptor(s) representing OS sockets to communicate with the master process. Everything sent over the socket would be pickled data.
In this slave process the module specified to run would have its __name__ sent to '__spawn__' or '__slave__' or something other than '__main__'. That way the same module could be used for either a master or slave process. In the built-in namespace there would also be an object just like in the master process that presented the connection between the two.
The call starting all of this should probably have an argument specifying if exceptions sent over the line should be raised immediately or just considered data. There should also be an atexit function registered to make sure that the processes are cleaned up properly.
Basically I want to have a command-line flag that takes in info stating where an interpreter should pick up some communication point between two processes on the same machine that works on UNIX and Windows. The spawned process has a special __name__ value to signify that it is a spawned process. The two processes communicate by sending pickled data back and forth. I guess this is kind of like a poor man's Erlang or fork().
