My little skunkworks website got cleared to be worked on, so I have shifted my focus to it. I think I am going to keep it a secret until
PyCon and announce it during a lightning talk for fun. But just because the subject of my work is a secret doesn't mean the process of creating the website needs to be kept from the public (even if it means I have to admit to what I am doing early; being secretive is not exactly critical).
I must admit that most of my experience in creating websites is relegated to either simple stuff or more JavaScript-heavy work. This is turning out to be the first website where there is a serious back-end that has offline workers, constantly updating data, etc. In other words it is a learning experience for me.
The first lesson is that I can't be sloppy with this website. Back on Sunday I started to slap together the website on
Google App Engine on my machine, simply coding away until I thought I reached a point where I could actually examine some data. I was using
task queues for the first time and trying to coordinate all of this communication with various queues running at various rates with different workers, etc. In other words I over-engineered.
But even worse is that I was not doing any proper testing. I figured I might just do some hand-driven tests on a subset of data to make sure things were working out. But my slap-dash coding quickly became a hindrance as I ran into several import errors thanks to me not paying attention and making sure to add the proper import statement. With the turn-around of launching a page which populated a task queue being long, I knew that I just couldn't keep going like this.
And so I knew I had to recode for testing. But how the heck do you test a website, let alone an App Engine website? I mean I know how to write proper unit tests, functional tests, etc., but I didn't know the best way to handle that for a website that has to run under some heavy infrastructure that has to be available.
This called for some research to even get tests executing. I stumbled across
nose-gae which seemed to give me what I needed to get the tests executed if I was willing to use
nose (which I was). I even tried to be a good little developer and work under
virtualenv, but I ran up against odd path issues involving App Engine that I simply did not want to try to diagnose after an attempt to use
gaeunit left me wanting (not enough traceback info to debug well). Luckily nose and nose-gae were the only things I felt the need to actually install so I didn't totally screw up my Python 2.5 install.
With tests runnable, I then had to decide how the heck I was going to do this. And that's when I decided I needed to refactor the hell out of my code. Being an App Engine app that is (at least currently) using no web framework, every URL resolves to a
request handler class. That means there is at least a get() or post() method on each class. So I made the decision that all request handling (e.g., getting what is in some POST argument but not even decoding the JSON) and all response handling (e.g., writing back out any HTML) would be handled in the get/post method, but nothing more; essentially they are gutted to the point of simply stitching together method calls, as one typically should when there is I/O involved. I then broke code down into helper methods as necessary in order to think about the functionality in the way I needed to. I also put all
transaction code in its own method since I am purposefully keeping that work to a computational minimum. This allows me to
mock out things like task queue usage in order to verify what I want to happen is going on without worrying about App Engine actually using some task queue and having to clean that up manually later (handling actual
datastore work is no big deal so I am happy to let that actually occur with proper test cleanup).
All of this leads to me writing abstracted, modular code where I can test non-network, non-App Engine code quickly and easily and then purposefully cleaning up or mock out as needed when I do need to work with App Engine's assets. At this point I have not written tests for the actual website page requests as the get()/post() methods as they are dirt-simple and they are just being called by App Engine task queue events (but eventually I will once I decide how I want to handle that specific case;
webtest?).
What is the lesson in all of this? Don't be sloppy. Assume you will have to test your code eventually, forcing you to write code in a fashion that makes the code easy to test. And then actually bother to test your code. Dealing with the network or APIs that have no way to undo or cancel an event programmatically (e.g., something in a task queue) should not stop you from writing tests as there are ways to isolate these things such that they are not a burden.
