Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)
2012-01-26
2012-01-24
Grab bag of tips when working with App Engine unit testing
[update: App Engine doc bug is being fixed and should be publicly visible in 1.6.2 or 1.6.3]
First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.
With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.
First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.
With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.
2012-01-17
Working with App Engine backends
This post is all about backends in App Engine, but first I want to make two personal points.
One is that I'm sorry I have not been updating this blog (or really contributing to Python and its great community) more this past year. It was a rather insane 2011 for me personally: I finished my Ph.D., got married, moved to SF (w/o my wife thanks to immigration), started working at Google full-time, moved to Toronto (yes, I moved internationally within the span of 6 months), transferred to the Google Waterloo office and started on a new team. In other words I have been stressed out and busy and busy continuously for what feels like ages. But now that I am back with my wife and I am done with the crazy international moving I don't expect 2012 to be anywhere near as nuts (nor any other year in the near future for that matter if I can help it).
Two, I am no longer on the App Engine team, so all that I am about to say (and will also say in the future as I have a couple of posts to write on App Engine) is from me as just another person using the service. I have been working on a redesign of a website of mine in my spare time and these next couple of posts are based on that work (if you following me on Google+ you know what it is, but I'm not ready for a full-blown Internet debut so I'm not going to link here yet). I don't have any insider knowledge here nor speak for the team.
One is that I'm sorry I have not been updating this blog (or really contributing to Python and its great community) more this past year. It was a rather insane 2011 for me personally: I finished my Ph.D., got married, moved to SF (w/o my wife thanks to immigration), started working at Google full-time, moved to Toronto (yes, I moved internationally within the span of 6 months), transferred to the Google Waterloo office and started on a new team. In other words I have been stressed out and busy and busy continuously for what feels like ages. But now that I am back with my wife and I am done with the crazy international moving I don't expect 2012 to be anywhere near as nuts (nor any other year in the near future for that matter if I can help it).
Two, I am no longer on the App Engine team, so all that I am about to say (and will also say in the future as I have a couple of posts to write on App Engine) is from me as just another person using the service. I have been working on a redesign of a website of mine in my spare time and these next couple of posts are based on that work (if you following me on Google+ you know what it is, but I'm not ready for a full-blown Internet debut so I'm not going to link here yet). I don't have any insider knowledge here nor speak for the team.
2011-09-12
PyCon 2012 CFP ends in a month!
The title says it all: the CFP for PyCon 2012 ends in a month, so you should seriously start thinking about getting those proposals in! As I say every year, you do not need to be a Python celebrity or have presented previously in order to get a talk accepted. Having been on the PC from the beginning of PyCon I can tell you that we very happily give slots to people we have never heard of. As long as your proposal shows you are organized and your topic is interesting your talk will be given a serious chance for acceptance.
And sorry about the lack of posts. My Python free time is limited at the moment as most free time is spent talking with my wife over a G+ Hangout thanks to US immigration perpetually dragging their feet, keeping her in Toronto and me in SF and causing general misery spread across the continent. That means what little time I do have I spend on trying to get importlib bootstrapped (which you can follow in my sandbox/bcannon repo in the bootstrap_importlib branch) instead of blogging. But once the bootstrapping is done (or I come up with some rather clever thing) I will do a technical post.
And sorry about the lack of posts. My Python free time is limited at the moment as most free time is spent talking with my wife over a G+ Hangout thanks to US immigration perpetually dragging their feet, keeping her in Toronto and me in SF and causing general misery spread across the continent. That means what little time I do have I spend on trying to get importlib bootstrapped (which you can follow in my sandbox/bcannon repo in the bootstrap_importlib branch) instead of blogging. But once the bootstrapping is done (or I come up with some rather clever thing) I will do a technical post.
2011-07-16
How to import a module from just a file path
Dr. Brown asked on Twitter whether there was a single expression (e.g., no semi-colons or abuses of and or or) that could import a module from a file path, either through a stdlib function call or just constructed from scratch; mod = import_from_path('some_file.txt'). Because it involved import, I got cc'ed on the tweet while various people tried to come up with a solution. In the end people realized that it's not possible in Python 2 (but it is in Python 3). But how hard could it be, right? Right?!?
2011-06-28
My personal plans for Python 3.3
Now that my life has hit another purgatory moment (waiting to here when my wife's permanent residency interview will be and still in the ramp-up period at work), I have had some time to think about what my personal goals will be for Python 3.3. This list is in no particular order and I make no promises to actually do any of it as I only have roughly 249 days until alpha1. =)
2011-04-22
Framerates for movies and the debacle that they are
One of my great weak points is technology, especially those involving movies. While I wouldn't go as far as calling myself a cinephile (e.g., I don't know every random fact about my favorite movies), I do love watching films which means I like trying to make my viewing experience at home rather nice (which honestly has not been that great thanks to my long-lasting student budget).
One of the latest things being debated about the quality of cinema is the frame rate of films. Currently movies project at 24 fps. ATSC (digital broadcasting version of NTSC) is typically 30 fps at 1080p although it can also do 24 fps and the funky rates film and old NTSC video show at (PAL is 25 fps but I'm North American, so I am going to stick with what I know in this blog post). Blu-ray can do 24 fps at 1080p. So everything these days can handle the native projection rate of 24 fps of 35 mm film.
But really, 24 fps?!? Any hardcore gamer is going to tell you that frame rate is unacceptable. You want at least 30, if not 60 fps. As it turns out, so do people watching movies. This is why most decent TVs these days at least output 120Hz video which helps to artificially raise the frame rate and lead to a smoother picture, no matter if the original material is 24 or 30 fps (or even 60, but that is typically reserved for video game output). And even newer TVs can do 240Hz, which leads to an even smoother experience (supposedly; have not experienced it myself).
But what about in the theater? Everything is still filmed and made for 24 fps and that is what projectors (digital and analog) use. Well, two of the more cutting edge directors currently working want to change this. James Cameron has talked about filming Avatar 2 & 3 at 48 or 60 fps. Peter Jackson has gone beyond saying and is actually doing by filming the Hobbit at 48 fps. According to Jackson the higher frame rate not only looks better, but it makes 3D something that is actually bearable to watch (i.e., the nasty flicker goes away). And it turns out most digital projectors can have their firmware upgraded to project films that use these higher frame rates. And with 48 fps being a multiple of 24, these directors could easily film at 48 fps and make 35 mm prints where every other frame is dropped so old projectors can still show at 24 fps.
But what about watching at home? While it's great that seeing those select films I want to see in the theaters will be projected at a higher frame rate (e.g., the Hobbit), what about when I want to watch at home? This is where that 48 fps gets sticky. Both blu-ray and ATSC do not support 48 fps at any resolution. Both do, however, support 60 fps at 720p. And I am willing to bet that Netflix streaming won't handle either 48 or 60 fps until we all have fiber connections. Another issue for the 48 fps is that it is not a common divisor of 120 (Hz), making it potentially nasty to artificially raise the frame rate; luckily 48 is a divisor for 240 (Hz).
So the question becomes, how does Hollywood decide to handle this theater/home frame rate discrepancy that is coming our way? Well, with this being Hollywood, I am willing to bet they do it in the way that screws over the home user the most, forcing you to go to the theater for the best experience. That would support the idea of going with 48 fps in theater projection. That way they still get a higher frame rate that helps out 3D projection, they can easily cut 24 fps versions for 35 mm prints, and they can stick with blu-ray and such at 24 fps as well. All of this while getting to keep a nice hook to get people who care about video quality to come to the theaters to pay for your expensive movie ticket. They can then come out with a new home video standard that supports 48 fps and force us to all buy our movie collections yet again (you know George would be happy to sell you Star Wars for the bazillionth time on a new format, although unless he lets Han shoot first I ain't going for it). If they go 60 fps they can at least do 720p60, but then again do we really want the drop in resolution?
One of the latest things being debated about the quality of cinema is the frame rate of films. Currently movies project at 24 fps. ATSC (digital broadcasting version of NTSC) is typically 30 fps at 1080p although it can also do 24 fps and the funky rates film and old NTSC video show at (PAL is 25 fps but I'm North American, so I am going to stick with what I know in this blog post). Blu-ray can do 24 fps at 1080p. So everything these days can handle the native projection rate of 24 fps of 35 mm film.
But really, 24 fps?!? Any hardcore gamer is going to tell you that frame rate is unacceptable. You want at least 30, if not 60 fps. As it turns out, so do people watching movies. This is why most decent TVs these days at least output 120Hz video which helps to artificially raise the frame rate and lead to a smoother picture, no matter if the original material is 24 or 30 fps (or even 60, but that is typically reserved for video game output). And even newer TVs can do 240Hz, which leads to an even smoother experience (supposedly; have not experienced it myself).
But what about in the theater? Everything is still filmed and made for 24 fps and that is what projectors (digital and analog) use. Well, two of the more cutting edge directors currently working want to change this. James Cameron has talked about filming Avatar 2 & 3 at 48 or 60 fps. Peter Jackson has gone beyond saying and is actually doing by filming the Hobbit at 48 fps. According to Jackson the higher frame rate not only looks better, but it makes 3D something that is actually bearable to watch (i.e., the nasty flicker goes away). And it turns out most digital projectors can have their firmware upgraded to project films that use these higher frame rates. And with 48 fps being a multiple of 24, these directors could easily film at 48 fps and make 35 mm prints where every other frame is dropped so old projectors can still show at 24 fps.
But what about watching at home? While it's great that seeing those select films I want to see in the theaters will be projected at a higher frame rate (e.g., the Hobbit), what about when I want to watch at home? This is where that 48 fps gets sticky. Both blu-ray and ATSC do not support 48 fps at any resolution. Both do, however, support 60 fps at 720p. And I am willing to bet that Netflix streaming won't handle either 48 or 60 fps until we all have fiber connections. Another issue for the 48 fps is that it is not a common divisor of 120 (Hz), making it potentially nasty to artificially raise the frame rate; luckily 48 is a divisor for 240 (Hz).
So the question becomes, how does Hollywood decide to handle this theater/home frame rate discrepancy that is coming our way? Well, with this being Hollywood, I am willing to bet they do it in the way that screws over the home user the most, forcing you to go to the theater for the best experience. That would support the idea of going with 48 fps in theater projection. That way they still get a higher frame rate that helps out 3D projection, they can easily cut 24 fps versions for 35 mm prints, and they can stick with blu-ray and such at 24 fps as well. All of this while getting to keep a nice hook to get people who care about video quality to come to the theaters to pay for your expensive movie ticket. They can then come out with a new home video standard that supports 48 fps and force us to all buy our movie collections yet again (you know George would be happy to sell you Star Wars for the bazillionth time on a new format, although unless he lets Han shoot first I ain't going for it). If they go 60 fps they can at least do 720p60, but then again do we really want the drop in resolution?
2011-03-23
importlib: doing it, and doing it, and doing it well
Earlier today I pushed a change to CPython which allows importlib to pass Python's test suite as the implementation of __import__ (sans failures because nothing expects __loader__ to be universally set on modules). This means that there is no known compatibility issues standing between me and making importlib CPython's implementation of __import__.
So what was the final hurdle? It basically boils down to code objects and their immutability. When you import from bytecode you are essentially loading a code object which represents the module (basically using marshal.loads()). The issue is that code objects embed what file they were created from (the co_filename attribute). But what happens if you relocate a .pyc file? Should co_filename point to the place the file was originally created at or at its current location? Turns out Python thinks it should be the latter.
The issue with updating the co_filename attribute on a code object is that it can only be done by C code; attributes on a code object are immutable from Python. My original plan was to modify marshal.loads() to take a filename argument which represented where the marshal data came from. That way marshal could fix co_filename while keeping the attribute immutable. But this didn't work.
It turns out that the attribute used by __import__ for fixing co_filename is not as thorough as the recursive solution I came up with for marshal.loads(). There is a loop that looks for objects to fix, but as soon as one of them is accurate the search is stopped by __import__. Rather than replicate this somewhat odd solution for marshal, I simply exposed a private API in imp for me to use in order to mutate a code object's co_filename attribute. Not an elegant solution, but since I am taking a PyPy view to import (i.e., it's my job to make importlib as compatible as possible and not tweak __import__ to fit my needs) I didn't have much of a choice.
Regardless, it feels very satisfying to have an implementation of import that passes the test suite now. This means I can work towards bootstrapping importlib as the implementation of import. I am planning to go with some bytecode-freezing solution for CPython where there is a build rule which recreates the bytecode any time importlib._bootstrap is mutated. That will remove the worry of having to rely upon some external file for import to work (otherwise I could come up with some import shim that did enough to import importlib directly). And with performance of importlib already being acceptable I feel fairly confident that this bootstrapping will happen in Python 3.3 (I have started work in the bootstrap_importlib branch of my personal repo).
So what was the final hurdle? It basically boils down to code objects and their immutability. When you import from bytecode you are essentially loading a code object which represents the module (basically using marshal.loads()). The issue is that code objects embed what file they were created from (the co_filename attribute). But what happens if you relocate a .pyc file? Should co_filename point to the place the file was originally created at or at its current location? Turns out Python thinks it should be the latter.
The issue with updating the co_filename attribute on a code object is that it can only be done by C code; attributes on a code object are immutable from Python. My original plan was to modify marshal.loads() to take a filename argument which represented where the marshal data came from. That way marshal could fix co_filename while keeping the attribute immutable. But this didn't work.
It turns out that the attribute used by __import__ for fixing co_filename is not as thorough as the recursive solution I came up with for marshal.loads(). There is a loop that looks for objects to fix, but as soon as one of them is accurate the search is stopped by __import__. Rather than replicate this somewhat odd solution for marshal, I simply exposed a private API in imp for me to use in order to mutate a code object's co_filename attribute. Not an elegant solution, but since I am taking a PyPy view to import (i.e., it's my job to make importlib as compatible as possible and not tweak __import__ to fit my needs) I didn't have much of a choice.
Regardless, it feels very satisfying to have an implementation of import that passes the test suite now. This means I can work towards bootstrapping importlib as the implementation of import. I am planning to go with some bytecode-freezing solution for CPython where there is a build rule which recreates the bytecode any time importlib._bootstrap is mutated. That will remove the worry of having to rely upon some external file for import to work (otherwise I could come up with some import shim that did enough to import importlib directly). And with performance of importlib already being acceptable I feel fairly confident that this bootstrapping will happen in Python 3.3 (I have started work in the bootstrap_importlib branch of my personal repo).
2011-03-20
PyCon 2011 wrap-up
tl;dr version: it was awesome! Read on to find out what I did at the sprints and two themes that I found at the conference.
2011-03-04
Secret site goes live: Python 3 Support on PyPI
When the Python 3 Wall of Shame launched a couple weeks ago, I (along with others) noticed discrepancies (discussed in the comments of the announcement blog post) between the real world and what the list said (e.g., doctutils was listed as not supporting Python 3). A similar issue came from On Python 3 Yet? Both websites seemed to be to accidentally spreading FUD about Python 3 support on PyPI thanks to the projects on PyPI not listing Python 3 support properly. While in no way intentional, it does cause issues when people use these sites as "proof" Python 3 still lacks support from the community.
So how did this happen? Well, projects on PyPI are listed as supporting Python 3 when they set the proper Python 3 trove classifier (instructions on how to fix this are in the Python 3 porting HOWTO). What I suspect has happened is that projects which do support Python 3 but don't have the proper classifier either (a) are not aware they should set it or (b) don't care/too lazy. Either way it leads to a mis-representation of what projects do and do not support Python 3.
To deal with this, I created a website to measure Python 3 support on PyPI. What differentiates this website from the others is that I am personally curating the list of projects that support Python 3. So while PyPI says docutils does not support Python 3, my website does. Same goes for projects which have forks that support Python 3 (e.g., setuptools "supports" Python 3 through distribute). I have even begun to flag projects which have working support in their version control system as "maybe" supporting Python 3. I completely understand why the creators of the other websites don't do this; it takes work. But since I have more invested in Python 3 than most people I am willing to put the work in.
If you go to the home page you will notice that I list a project as either supporting, not supporting, or maybe supporting Python 3. The latter case is there because a ton of projects do not specify at all if they even support Python 2! My hope is that projects begin to properly set their trove classifiers. This not only benefits the community by letting people know if they do or do not support Python 3, but because it can be used to know how far back they support Python. Take Django 1.2.5 as an example. It does not have trove classifiers set to let me know that it supports Python 2.4 through Python 2.7. Heck, I don't even know if they have tested against Python 2.7 yet. But if they set their trove classifiers this would be known both visibly and programmatically.
I am asking the community for help combating this problem of projects not listing their Python 3 support properly. If you know of a project which supports Python 3 somehow but that fact is not known on this new website, please either list it over on the Convore discussion on this topic, leave a comment here, or leave a comment on Google Buzz along with proof of the Python 3 support (e.g., a link to an official page stating the support). If you tack on a project's PyPI name on to http://py3ksupport.appspot.com/pypi/ (e.g., http://py3ksupport.appspot.com/pypi/docutils) you can see if the project's support has been picked up yet or not. As I said, forks do qualify for support as well as do projects which have functioning support in version control but have just not done a release yet. This is all rather important for the top projects as listed on the home page since that is what people will really pay attention to in order to notice Python 3 uptake.
To give people some scope of how much of a difference this curation makes, consider the top 50 projects by downloads for any release. 14 of the top 50 (i.e., 28%) support Python 3 somehow. But if you went by only trove classifiers you would think only 9 project (18%) supported Python 3. Even hard coding modules absorbed into Python's stdlib would leave off 2 of the projects. If you look at latest release downloads you have even bigger support and discrepancies: 16 out of 50 projects support Python 3. But of those, 8 I had to manually flag (4 of which are not from the stdlib). Percentage-wise this manual effort makes a difference and gives a much better indication of how much support there already is for Python 3.
So this is what I have been working on for the past two weeks. I learned a lot about App Engine and creating a scalable website. The whole thing updates every 20 minutes from PyPI in terms of project metdata changes. I also update download totals every day. Long term I hope to add more metrics to gauge what the top projects are (e.g., listed dependencies, Google Code searches, etc.). I also want to develop a Chrome extension which will let a user know when a project supports Python 3 when viewing PyPI even if it doesn't say so (maybe even getting to the point that people can notify me through a button click that the project actually does support Python 3). I also want an API so that people on other websites that use different metrics can have access to the same data I use to mark a project as supported or not. But as of right now, I'm ready to take the weekend off. =)
So how did this happen? Well, projects on PyPI are listed as supporting Python 3 when they set the proper Python 3 trove classifier (instructions on how to fix this are in the Python 3 porting HOWTO). What I suspect has happened is that projects which do support Python 3 but don't have the proper classifier either (a) are not aware they should set it or (b) don't care/too lazy. Either way it leads to a mis-representation of what projects do and do not support Python 3.
To deal with this, I created a website to measure Python 3 support on PyPI. What differentiates this website from the others is that I am personally curating the list of projects that support Python 3. So while PyPI says docutils does not support Python 3, my website does. Same goes for projects which have forks that support Python 3 (e.g., setuptools "supports" Python 3 through distribute). I have even begun to flag projects which have working support in their version control system as "maybe" supporting Python 3. I completely understand why the creators of the other websites don't do this; it takes work. But since I have more invested in Python 3 than most people I am willing to put the work in.
If you go to the home page you will notice that I list a project as either supporting, not supporting, or maybe supporting Python 3. The latter case is there because a ton of projects do not specify at all if they even support Python 2! My hope is that projects begin to properly set their trove classifiers. This not only benefits the community by letting people know if they do or do not support Python 3, but because it can be used to know how far back they support Python. Take Django 1.2.5 as an example. It does not have trove classifiers set to let me know that it supports Python 2.4 through Python 2.7. Heck, I don't even know if they have tested against Python 2.7 yet. But if they set their trove classifiers this would be known both visibly and programmatically.
I am asking the community for help combating this problem of projects not listing their Python 3 support properly. If you know of a project which supports Python 3 somehow but that fact is not known on this new website, please either list it over on the Convore discussion on this topic, leave a comment here, or leave a comment on Google Buzz along with proof of the Python 3 support (e.g., a link to an official page stating the support). If you tack on a project's PyPI name on to http://py3ksupport.appspot.com/pypi/ (e.g., http://py3ksupport.appspot.com/pypi/docutils) you can see if the project's support has been picked up yet or not. As I said, forks do qualify for support as well as do projects which have functioning support in version control but have just not done a release yet. This is all rather important for the top projects as listed on the home page since that is what people will really pay attention to in order to notice Python 3 uptake.
To give people some scope of how much of a difference this curation makes, consider the top 50 projects by downloads for any release. 14 of the top 50 (i.e., 28%) support Python 3 somehow. But if you went by only trove classifiers you would think only 9 project (18%) supported Python 3. Even hard coding modules absorbed into Python's stdlib would leave off 2 of the projects. If you look at latest release downloads you have even bigger support and discrepancies: 16 out of 50 projects support Python 3. But of those, 8 I had to manually flag (4 of which are not from the stdlib). Percentage-wise this manual effort makes a difference and gives a much better indication of how much support there already is for Python 3.
So this is what I have been working on for the past two weeks. I learned a lot about App Engine and creating a scalable website. The whole thing updates every 20 minutes from PyPI in terms of project metdata changes. I also update download totals every day. Long term I hope to add more metrics to gauge what the top projects are (e.g., listed dependencies, Google Code searches, etc.). I also want to develop a Chrome extension which will let a user know when a project supports Python 3 when viewing PyPI even if it doesn't say so (maybe even getting to the point that people can notify me through a button click that the project actually does support Python 3). I also want an API so that people on other websites that use different metrics can have access to the same data I use to mark a project as supported or not. But as of right now, I'm ready to take the weekend off. =)
Subscribe to:
Posts (Atom)