PyCon talk decisions have been made

[edit: nixed a paragraph by request]
[edit: link to accepted talks and thank Doug]

I want to say three things in this post. One is that the PyCon program committee has finished making there decisions of which 95 talks out of the 179 submissions we received. I should also admit that my talk (#9 in the list) was accepted although I actually missed the part of PC meetings where it was so I didn't officially find out until everyone else did.

Second, I wanted to publicly thank Jesse Noller and the other members of the pycon-pc for their hard work (with special honorable mention to Doug Napoleone for pycon-tech, the software that keeps PyCon running). We had a record number of submissions with an average of only 10 people in IRC to go over them during the culling process. It was grueling, but people stuck in there and helped make it happen (and I admit I did not participate as much as I would have liked due to a conference paper submission I had).

Third, I want to address the negativity that has been popping up about the decisions we had to make (some of which led to hate mail sent personally to Jesse which is completely uncalled for). There seems to be two themes that have popped up as to why people are upset over there rejection.

One is their talk received all positive reviews but was still rejected. Sorry, but that's called time restriction. We honestly had more talks with all positive reviews than we had slots. And just because your talk didn't receive any negative reviews does not mean that it fired anyone up enough to want to stand up for it. We use a Champion voting system where a talk only gets considered if a reviewer is willing to take a stand saying they will fight for the talk to be included. If a talk has a lot of people standing up for it then it will get in without question, but it takes A LOT of people for that to happen (like four or more, and there were not that many talks like that). Otherwise we have to discuss the talk. And if the people fighting for the talk can't convince the PC (or don't make the IRC meeting which is just life since we can't infinitely postpone to make everyone's schedule) then the talk is let go. So if your talk received all positive reviews and you are wondering what it was rejected in the end, it means it unfortunately didn't attract a champion that was either to argue for it in a way to win over the other PC members in IRC.

The second theme has been how the talks are not anonymous to reviewers. This is a conscious decision that we have made from years of experience where it matters whether we know upfront whether someone is a good speaker. We tried anonymous reviewing one year and it turned out badly. PyCon does not have a proceedings like academic computer science conferences where if someone can't present you still have the work in paper form. At PyCon if you are a good speaker that counts for a lot as speaking is how you share your knowledge. And if people think you are not a great speaker that usually means someone won't champion your talk, not that you will get a -0 or -1 (if that does happen it is usually a sign of a bad review, but once again we were short on people and some bad reviews slipped through).

Submit a poster for PyCon 2010

The PyCon organizers are trying something new for PyCon 2010: posters. Basically there is a plenary session where people have posters about something and they stand there talking to whomever comes by (if you have been in academia you know what these can be like). In my time doing posters at academic conferences I have found that at the larger conferences a poster can be worth your time. You end up with enough people coming by talking to you that you get valuable feedback and contacts. Plus a poster is a lot less stressful to do compared to a full talk and allows for more detail than a lightning talk.

So if you have something you want to say that can be put on a poster and want some feedback on it or simply share with the PyCon attendees, please consider making a poster.

I ♥ jQuery (UI)

I had this grand plan for a blog post comparing the various JavaScript GUI libraries, which in the end ended up being this blog post stating how I like jQuery UI. If you don't care about reading a "love letter" to jQuery UI, then you can stop reading now.

Everyone should switch to Distribute and off of setuptools

[edit: mention how the fix in Python 2.6.3 may have not belonged in 2.6.3 since it is a micro release]

Python 2.6.3 has a couple of bugs still lingering that warrant a brown bag 2.6.4 release (should be out before the end of the month). One of the "bugs", and the entire reason I am doing this blog post, involves distutils and setuptools. Turns out that for Python 2.6.3 a change was made to distutils that broke setuptools for building extension modules. While the change that broke setuptools is being viewed as improper for a micro release (code shouldn't break in a micro release unless it was really bad semantics being fixed), it did bring to my attention that a lot of people do not know about Distribute.

A problem is that setuptools is no longer maintained. Luckily there is already a solution to this predicament that the wider Python community might not be fully aware of.  Tarek Ziadé forked setuptools and created Distribute, with its first release two months ago, explicitly to provide a library compatible with setuptools that is being actively maintained (and thus has bug fixes). Distribute is a drop-in replacement for setuptools, complete with being able to import it under the setuptools name so that everything will continue to work as if you had setuptools itself installed sans some bugs.

Because Distribute fixes bugs and is a backwards-compatible drop-in replacement I HIGHLY encourage people to change their installs of setuptools to Distribute. Everything will continue to work as Distribute installs itself under the setuptools name to maintain backwards-compatibility. Gentoo has even switched. Plus Distribute 0.6.3 supports Python 3.

So please, if you use setuptools then upgrade to Distribute.

Adding structural consistency to exceptions

When I wrote PEP 352: Required Superclass for Exceptions one of the hopes I had was it would lead to some sanity when it came to attaching information to exceptions. As it stands now everything is stored under BaseException.args which is not exactly structured, and so I wanted to clean that situation up. Unfortunately this got derailed because of backwards-compatibility issues (and for those of you being bitten by the BaseException.message deprecation, Python 2.6.1 has a fix to make it saner).

But today a tweet from Jacob Kaplan-Moss reminded me why exceptions need to be cleaned up. If you look at the constructor for BaseException it is essentially:

def __init__(self, *args):

  self.args = args

Not exactly fancy, but this loose generality has a price in that most people simply toss in all potentially useful information into the exception constructor without providing a way to get any information out of it in a reasonable way. For instance, if you have arguments 2 and 3 contain useful information why do I have to know what index provides what info instead of using a descriptive attribute name? If I can't use dir() on an exception to figure out what useful metadata is on an exception then there is a problem. There's a reason named tuples came into existence; indexes are not self-documenting.

There is also a nastier side-effect for exceptions given multiple arguments in regards to how BaseException.__str__() acts. If args has exactly a single value then the string for the exception is str(args[0]). But if args has multiple values then the string of the exception is str(args). That can lead people not tacking on any information to make sure their exceptions have a nice, clean string representation. That's what leads people like JKM having to parse data out of an exception's string. It's just ludicrous for anyone to have to parse an exception to get information that was obviously available when the message was created!

I see two solutions to this predicament, and both involve changing the constructor to BaseException. Let's look at each in turn and use IndexError as an example of how things might improve where we attach to the exception what index was out of what range. So one option is to simply change BaseException to only accept a single argument and have that be bound to message and be what the string representation is:

class BaseException:

    def __init__(self, message=''):

        self.message = message

    def __str__(self):

        return str(self.message)

That would motivate me to change IndexError to:

class IndexError(Exception):

    def __init__(self, message="index out of range", *, index=None, range=None):

        if index is not None:

            message = "index {index} out of range"

            if range is not None:

                message += " {range}, exclusive"

        super().__init__(message.format(index=index, range=range))

        self.index = index

        self.range = range

By having BaseException accept only a single argument people who typically just toss metadata about why the exception is being raised are forced to actually construct a message. The hope is that if someone is being force to construct a message for an exception on their own they will simply tack the information on to the instance.

But what about those exception authors who are fine creating a message but still don't bother to tack the metadata on to the exception? That brings up the other possible approach to BaseException where it does string interpolation for you based on keyword arguments you pass in:

class BaseException:

    def __init__(self, message, **kwargs):

        self.message = message.format(**kwargs)

        self.__dict__.update(kwargs)

    def __str__(self):

        return str(self.message)

With IndexError we can now have a couple of options. One is to simply subclass and hope that most people will simply do IndexError("index {index} out of range", index=42) in all instances. The other option is to take the approach shown above but cut out some code that is no longer needed:

class IndexError(Exception):

    def __init__(self, message="index is out of range", *, index=None, range=None, **kwargs):

        if index is not None:

            message = "index {index} out of range"

            if range is not None:

                message += " {range}, exclusive"

        super().__init__(message, index=index, range=range, **kwargs)

This approach has the nice effect of promoting people to simply let BaseException construct the message string for the user while providing the nice side-effect of also storing the data on the exception using descriptive attribute names. It also allows for easy arbitrary metadata through kwargs more than the previous approach where you would have to explicit take kwargs and then update the instance.

Regardless of the approach, the real trick is how would you transition over to it? First thing would be to introduce a pending deprecation for BaseException taking more than a single argument. Next would be to activate the new semantics with keyword arguments where if more than a single positional argument is given an exception is thrown. After a certain amount of time

But the real trick for a transition is message. Do you only set it for exceptions that have transitioned? If you do set it for exceptions that are still passing in multiple positional arguments do you set the attribute to the first argument or the string representation for all of them? Or do you simply skip having message and just let people call str() on exceptions to get what the message would be, like so?

class BaseException:

    def __init__(self, message, **kwargs):

        self.__message = message

        self.__dict__.update(kwargs)

    def __str__(self):

        return self.__message.format(**self.__dict__)

I honestly don't know what the best solution would be. Some people would probably complain about not being able to introspect on the message attribute, but exposing a string format seems somewhat icky to me. Either way some solution would be available.

Who knows, maybe some day I will try to push this into Python and finish what I had originally intended to do with PEP 352.

PyCon 2010 talk proposals due in two weeks

Just a reminder to folks out there that the due date for submitting a talk to PyCon 2010 is two weeks away!

Evolving the standard library

As Titus blogged, an interesting discussion was started over on the stdlib-sig about whether argparse should be added to the standard library, and if so how to handle/whether to deprecate getopt and optparse. Since the discussion showed rather well how people think the standard library should evolve I figured I would blog about those views and my own (assuming I don't mess up and misrepresent people
=).

Intersection of built-in modules between CPython, Jython and IronPython

[EDIT: updated for IronPython 2.6b2; made it clearer which VMs are missing what modules that importlib relies upon]

It has been a big goal of mine to make importlib the default implementation of import for CPython. But an even bigger goal has been to make it the default implementation for ALL full featured implementations of Python once they implement Python 3. Not only would it make sure that all VMs have consistent semantics when it came to imports, but to also prevent every VM from having to re-implement import themselves.

But using importlib as import imposes a bootstrapping problem. How do you import, well, import? First off, you need to find the source code, compile it into a code object, and create a module object using that code object. That part is actually easy as you can simply look for the file on sys.path since you know what you are looking for, you can compile the source using the built-in compile() function, and then you finally create a module and initialize it with exec(). This is essentially what importlib does at a rudimentary level.

But import obviously goes beyond the rudimentary. There is bytecode to read and write, packages to deal with, warnings to raise, etc. And all of that requires code from some module in the standard library. But if you are trying to bootstrap in import w/o having a full-featured import, what do you do? You rely on built-in modules is what you do.

By using built-in modules you could have the VM inject any built-in module into the created importlib module and have it begin using it. Because of this I was curious as to what built-in modules CPython 3.1, Jython 2.5, and IronPython 2.6b2 had in common. The results are:

• _codecs
• _functools
• _sre
• _weakref
• errno
• gc
• imp
• sys

Not a whole lot. Importlib itself relies upon:

errno

Everyone has this.

io

IronPython's _bytesio probably has what I need (importlib only uses io.FileIO). Jython does not cover yet 2.6 so there is hope.

imp

Everyone has this.

marshal

This is actually optional (or at least I will make sure it is) as VMs do not need to implement pyc support.

posix/nt/os2

IronPython has this. Jython plans to have this in 2.6.

sys

Everyone has this.

warnings

Jython does not have a native implementation, but importlib only needs warnings.warn().

There is a partial overlap, but not a complete overlap. Luckily this is for Python 3 and thus there is hope that some of the things I need can be made common between the VMs in terms of what the built-in modules provide. It's possible that IronPython has everything already and Jython could add only what importlib needs (probably) w/o much issue.

Otherwise I am causing myself more pain than I need to and I should just not worry about the bootstrap and simply import code directly. Copying code from the 'os' module does get a little annoying after a while. =)

Less than a month to submit a PyCon 2010 talk

PyCon talk proposals are due October 1, which is less than a month (four weeks) away. I have already submitted a talk on custom importers and using importlib to write your own.

Compiling Python using Clang

[edit: added compilation timings]

Like many people (if Twitter is any indication), I upgraded to Snow Leopard and XCode 3.2 this past weekend. One of the nice things that came with the new Developer Tools is Clang 1.0. I have been anticipating the stable release of this tool ever since I watched a video from the LLVM conference on it over a year ago. With it's much improved warning output compared to gcc and it's faster compilation time I wanted to give it a try on CPython.

First off, though, credit needs to be given to the Unladen Swallow guys, and especially Jeffrey Yasskin, for working out some nasty bugs that used to prevent LLVM from compiling CPython over the past year. Without the fixes I would have just given up on using clang.

With CPython now cleanly compiling with clang, I decided to give it a spin. The environment variables I ended up using specific to clang were:

• CC = clang
• CFLAGS = -Qunused-arguments
• CPPFLAGS = -Qunused-arguments

The "-Qunused-arguments" flag tells clang to not complain if it is given command-line arguments that are redundant or unused. If you don't do this you can end up with a ton of warnings about unneeded CPPFLAGS arguments. And it is used in both CFLAGS and CPPFLAGS as otherwise it isn't picked up when setup.py runs (I don't think setup.py or distutils uses CFLAGS at the moment). But otherwise CPython builds fine!

One other thing you might want to try using when building CPython is "-Wno-unused-value". It turns out that PyObject_INIT() and PyObject_INIT_VAR() never have their returned values used explicitly and this flag turns off those warnings as there are a bunch of them and each one refers to two other code locations.

After I originally posted this I got one comment here and a couple on Twitter about what the benchmarking timings were. I caved in and ran them with ``/configure --prefix=/dev/null --with-pydebug --with-computed-gotos --with-universal-archs="64-bit``. In Clang it took a total of 36 seconds while with gcc 37 seconds. So the speed increase is minimal, but the important thing to remember is that the debugging information that Clang spits out is far and away better than what gcc gives you. So while the performance difference is small, the debugging output are not even close to being equal in terms of readability.