2012-01-26

Asynchronous XML-RPC in Python

Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)

In my case I needed to communicate with an XML-RPC server that supported multi-calls, but in an asynchronous fashion with a non-standard communication object (i.e. not sockets but App Engine's urlfetch). So I had to piece together how to make a XML-RPC client call that used multi-calls so that I could do it asynchronously myself.

The first thing to know is that a multi-call is a system.multicall function call to the server. It takes a sequence of arguments that specify what each individual function call should be. Each sequence item contains a dict with two keys: "methodName" and "params". The "methodName" key has a string value of the function name to call on the XML-RPC server. The "params" key holds the tuple of arguments (and it must be the tuple type and you must use a tuple even if it is just a single argument --think of what it would take for the call to work with *args) to pass to the function named in "methodName".

Knowing what function you are calling on the other end (system.multicall) and having a sequence of dicts specifying what you are calling in hand, you can make your call to the server. First you need to encode everything into XML: xmlrpc.client.dumps((seq_of_args,), 'system.multicall') (notice how I passed the sequence of dicts in a tuple -- and I mean a tuple, not a list). With the XML now in hand, you make the actual call to the server by making a POST with the XML as the body and set the Content-Type header to text/xml using whatever asynchronous mechanism you want to use to make the call.

What you get back from the server is XML that you can load with xmlrpc.client.loads(data, use_datetime=True)(chances are you will want the [0][0] value as that contains a sequence of results from each function call you requested). Do realize, though, that there is no indication of what function returned what, so you will need to correlate the index of the return value with what call you made (zip() makes this trivial).

And that's how you construct an XML-RPC multi-call from scratch in Python so you can make an asynchronous call. If you want to see an example of all this, see this code.