2008-11-22

Using the AST and decorators for L10n translations

I had a thought the other day about L10n and how I have never thought the gettext approach was optimal. The idea of having to wrap every string literal in a call to the '_' method seems needlessly sub-optimal. What happens if you forget to wrap a string? What about the overhead of having to call that function for every string literal used by your code? And don't you have to run your app completely to create the translation corpus, hoping every string literal is exercised?

There has to be a better way for both creating the corpus and handling the translation. For the corpus, I think the AST could be used to extract every non-docstring string literal from code. Simply compile your Python source to an AST and you can walk it to find all string literals. You also have the perk of knowing exactly where a string came from since the AST is annotated with that kind of information. Since it is done statically it doesn't rely on testing to give 100% coverage of your string literals (which you should have anyway, but we all know the world is not perfect).

As for translation, I see to possibilities based on whether you want to allow for translations between executions. If you only care about installations supporting only a single translation, you can take the AST idea and simply rewrite the AST such that string literals are replaced with their translations. Then you simply write out the translated AST to bytecode. As long as the bytecode is always used there is absolutely no overhead in using a translation of your application.

But that seems rather heavy-handed. This is where decorators come in. Let's say you decorate every function that contains a string literal. I know this doesn't solve the same issue of the '_' function accidentally being left off, but a function decorator is required to be put in fewer places than every string literal. With this decorator in place, it checks to see what the locale is. If it is different than what the string literals are, it takes the function object and re-creates the code object with a new co_consts that contains the translated strings. Now your overhead is simply the creation of new function objects which should be no worse, and is probably better, than calling a function for every string literal.

And you can have this translation be a one-time cost or fully dynamic per execution. If you simply have the decorator return the new function object then the translation is done at function creation time and is never worried about again. But if you instead return a closure that checks per run-time you can figure out if you need to create a new function object for every call. Since people tend not to change their locale in the middle of execution constantly it should be no more expensive than finding out the current locale and checking a variable to see if they differ, so it's still not very expensive.

But what if this idea is ridiculous and you still want to use gettext and its approach? Well, the AST trick can still be used to generate the corpus. But it can also be used to verify that every string literal is wrapped in a call to '_'. If it isn't you can simply raise an error in your build step and add the appropriate call.

Now since this is just an idea I had the other day and I am not a huge user of L10n, I have not researched this in any way. It is possible someone has already thought of this and has a package out there that does this exact thing, which would be neat as it means my idea is not completely ludicrous.