2011-03-04

Secret site goes live: Python 3 Support on PyPI

When the Python 3 Wall of Shame launched a couple weeks ago, I (along with others) noticed discrepancies (discussed in the comments of the announcement blog post) between the real world and what the list said (e.g., doctutils was listed as not supporting Python 3). A similar issue came from On Python 3 Yet? Both websites seemed to be to accidentally spreading FUD about Python 3 support on PyPI thanks to the projects on PyPI not listing Python 3 support properly. While in no way intentional, it does cause issues when people use these sites as "proof" Python 3 still lacks support from the community.

So how did this happen? Well, projects on PyPI are listed as supporting Python 3 when they set the proper Python 3 trove classifier (instructions on how to fix this are in the Python 3 porting HOWTO). What I suspect has happened is that projects which do support Python 3 but don't have the proper classifier either (a) are not aware they should set it or (b) don't care/too lazy. Either way it leads to a mis-representation of what projects do and do not support Python 3.

To deal with this, I created a website to measure Python 3 support on PyPI. What differentiates this website from the others is that I am personally curating the list of projects that support Python 3. So while PyPI says docutils does not support Python 3, my website does. Same goes for projects which have forks that support Python 3 (e.g., setuptools "supports" Python 3 through distribute). I have even begun to flag projects which have working support in their version control system as "maybe" supporting Python 3. I completely understand why the creators of the other websites don't do this; it takes work. But since I have more invested in Python 3 than most people I am willing to put the work in.

If you go to the home page you will notice that I list a project as either supporting, not supporting, or maybe supporting Python 3. The latter case is there because a ton of projects do not specify at all if they even support Python 2! My hope is that projects begin to properly set their trove classifiers. This not only benefits the community by letting people know if they do or do not support Python 3, but because it can be used to know how far back they support Python.  Take Django 1.2.5 as an example. It does not have trove classifiers set to let me know that it supports Python 2.4 through Python 2.7. Heck, I don't even know if they have tested against Python 2.7 yet. But if they set their trove classifiers this would be known both visibly and programmatically.

I am asking the community for help combating this problem of projects not listing their Python 3 support properly. If you know of a project which supports Python 3 somehow but that fact is not known on this new website, please either list it over on the Convore discussion on this topic, leave a comment here, or leave a comment on Google Buzz along with proof of the Python 3 support (e.g., a link to an official page stating the support). If you tack on a project's PyPI name on to http://py3ksupport.appspot.com/pypi/ (e.g., http://py3ksupport.appspot.com/pypi/docutils) you can see if the project's support has been picked up yet or not. As I said, forks do qualify for support as well as do projects which have functioning support in version control but have just not done a release yet. This is all rather important for the top projects as listed on the home page since that is what people will really pay attention to in order to notice Python 3 uptake.

To give people some scope of how much of a difference this curation makes, consider the top 50 projects by downloads for any release. 14 of the top 50 (i.e., 28%) support Python 3 somehow. But if you went by only trove classifiers you would think only 9 project (18%) supported Python 3. Even hard coding modules absorbed into Python's stdlib would leave off 2 of the projects. If you look at latest release downloads you have even bigger support and discrepancies: 16 out of 50 projects support Python 3. But of those, 8 I had to manually flag (4 of which are not from the stdlib). Percentage-wise this manual effort makes a difference and gives a much better indication of how much support there already is for Python 3.

So this is what I have been working on for the past two weeks. I learned a lot about App Engine and creating a scalable website. The whole thing updates every 20 minutes from PyPI in terms of project metdata changes. I also update download totals every day. Long term I hope to add more metrics to gauge what the top projects are (e.g., listed dependencies, Google Code searches, etc.). I also want to develop a Chrome extension which will let a user know when a project supports Python 3 when viewing PyPI even if it doesn't say so (maybe even getting to the point that people can notify me through a button click that the project actually does support Python 3). I also want an API so that people on other websites that use different metrics can have access to the same data I use to mark a project as supported or not. But as of right now, I'm ready to take the weekend off. =)