More fiddling with Django and Google App Engine

After having done a static page with some jQuery for my home page, and then a very simple web app to manage the list of TV shows I watch (so I know how far behind I am in watching them), I figured the next web app should be slightly more complicated.

So I tackled lists in a generic fashion. I don't know where I get it, but I tend to make lists of stuff. Nothing significant, mind you, but I do have several lists that I am constantly updating in my head on various topics, such as stuff I want to do in Vancouver before I graduate and leave and what I would do if I was given $20,000,000.

And so I tried to come up with a generic web app for unsorted and definition lists. Now when I have done my web apps I have tried to stick to the DRY principle as much as possible. This meant that I only needed to define a single list model as no matter what kind of list you had, a list is a list at its most basic level. But what type of list does control what information you want per entry of that list. That means two models for list entries. That's slightly tricky since list entries need a reference to the list they are a part of, and thus leads to two different reference sets under Google App Engine's datastore model.

So I did what any programmer would do in this situation and I abstracted with a dash of information upfront. When a list is created, I specify what kind of list it is since that does not fluctuate. Based on that information, various methods defined on the list model pull list entries from the proper reference set (e.g. if a list is defined to be a unordered list it has the items() method return from the unordereditem_set attribute on the list model). And I continued to abstract everything I needed on the list model to the point that I use the list model to get the list entry class that is appropriate. I basically work blindly in the code as to what list entry model is being used.

The other thing I did was define only a single management view for lists and list entries. This means that all creation and editing of lists goes to a single view (same goes for list entries). This allowed me to control more based on URLs than I do from my TV series web app. It does lead to more branching within the single management view, but it keeps things simpler for me as any commonality is easily handled. Besides, I prefer to have a single method that handles a single class of issues based on its arguments more than defining a bunch of custom methods.

And Django deserves credit for letting the views be simplified like this. Being able to generate the form to use is generic enough that I was able to use my abstraction setup for the list types to not have to care what model form I was using; I just used ``list.item_class().form`` and just didn't care.

And the last improvment I did was memcache the hell out of my entire site. I defined a subclass of the Client class provided by App Engine. It allows me to take my old memoize decorator and have it generate the key to use automatically based on the function being decorated and a common prefix. I even went as far as to provide a key argument that is a callable that can return parts to use in the key so that it can be dynamic based on the arguments to the decorated function (which is apparently a complaint someone had with my simple decorator I posted at the App Engine cookbook site). Although, in hindsight, it might be simpler and be a stronger common-case to do it based on the URL instead (sans GET arguments) for the common case instead of the view function's name. Then using Django's reverse() would be all that is needed to figure out what to delete from the memcache. Ugh, why didn't I think of this simplification last night?

When I add or edit something now I just delete the proper entry in the memcache. And if someone is logged in as an admin then no caching is used since admin-specific stuff is put in the HTML.

Once again, I am a happy camper when it comes to App Engine and Django. And once the Django helper for App Engine releases a 1.0 version and I can run that and Django from a zip file, I will a REALLY happy camper. I think the only thing really missing that I wish I had was a slug property for models so that I didn't have to specify them myself by hand. That, and a method I knew I could override that was always called upon commitment to the datastore.

First dynamic section to my web site is up

I finally got around to constructing the first dynamic part of my web site: a system for me to record what television shows I have watched from their pilot episode. While not designed to be a web app used by the world, I am able to add end edit any series listed. And the list that is there is not exhaustive; it's mostly so I know what shows I am behind in watching.

I started with this app because it was simple. The data schema has no relations so I didn't have to worry about doing any many-to-one relations (but another app I have planned will). Probably the trickiest part was setting a 'sort_on' field for TV titles that start with an article ('A', 'An', and 'The'). I wanted that to be based on the specified title for the series, so I overloaded the __init__() method for the model and set it manually. With a subsequent super() call the model class doesn't know that a required value was computed on the fly.

I did go back on forth on whether to use None values for season/episode to represent that I have watched an entire series, but decided against it. I figured it was better to be able to require a value for those attributes and just have an explicit 'completed' boolean attribute.

And I am not happy with how I am currently handling my templates. I am having a tough time deciding how much should be passed into a template through a view, what should be set through a block in a template, and what should just be a a global variable/filter. A good example of this stuff from Google App Engine's Users API. There are functions to specify login and logout URLs along with functions that return what user is logged in and whether they are an admin.

Right now I explicitly set 'user_is_admin' for my templates through the view to decide if a page should display admin-specific info (e.g. edit links by the names of the TV series). But since what user is logged in and whether they are an admin is a project-side thing I feel like writing a Django context processor to handle this. I could then provide a filter that generated login/logout URLs with default to the current URL (although that would probably require a RequestContext to work). Basically I am still feeling around to find out what the right abstraction level is for templates.

Adding a custom 404 page in Django

With my web site consisting of a single page, I suspect some 404s are going to get served rather quickly. So I decided to go ahead and do a custom 404 page. Google Webmaster Tools has a handy 404 widget you can paste into your 404 page that gives suggestions on where the person probably meant to go, along with a site-specific search box. Google Analytics also has some custom 404 code to help you track the bad pages.

And all of this was done by just defining a 404.html template in Django. The only problem I hit was that I couldn't have nested blocks for my Google Analytics code. The 404 code between a normal page view and the 404 are arguments to a single JS function. I wanted the Google Analytics code to be in a block and then have a nested block for the arguments to the JS function. In the end I just stripped out the Google Analytics block and just went with the argument block. Hopefully I won't come across a need to not track a page.

Quick review of "Practical Django Projects"

James Bennett was nice enough to have Apress send me a free copy of his book 'Practical Django Projects' after a blogged asking for help on how best to learn Django. To be totally upfront about this, while James had me sent a free copy, he did not ask for a public review, and James offered me the copy, not his publisher.

I am happy with the book. I feel like I have a reasonable grasp now on what Django provides me and how to structure my site and apps so as to maximize Django's abilities. Granted, I do plan to use Google App Engine and thus a lot of the tricks with the models won't work for me, but thanks to the book I still plan to use Django for implementing views and control code (i.e., everything you can do without using Django's ORM) for my site.

What is the best way to learn Django and web development?

Dear Lazyweb,

With PEP 3108 nearing completion, it is time to start thinking about finally developing my personal web site (so that I can then redo my father's business web site). That means I need to learn web development in general, and Django in particular.

But how to begin? I was planning on reading the Django book, but is there something else I should also read?

And how about web development in general? I believe Django comes with a testing framework, but is there something else I should be using? Any common design pitfalls I should be looking out for?

I am not looking to do fancy web applications. I just want the skills necessary to take information, both entered directly into the web site or from external files, and display it online. The goal is to be able to easily redo my father's static web site such that he can log into the site's admin page and add/edit content directly without me having to walk him through it over the phone (in other words my dad can browse the web but I am not about to ask him to write raw HTML or something).

Anyway, any help that people can lend me would be greatly appreciated.

My first (real) experience with XML (through ElementTree) and Django Templates (through Jinja)

After asking people for advice about how to handle open source work in one's curriculum vitae, I wrote an app to help me manage my CV. I decided to use XML as the data format and then transform it somehow from XML into various other formats (text, HTML, PDF through LaTeX). Nothing fancy, although it was a first for me in terms of serious XML use from my own schema design.

To start, I wrote a rough XML schema using RELAX NG Compact syntax. This was not too bad, although I did have a battle myself over how often to use attributes compared to a full-blown tag. I also cheated in some spots and didn't properly specify when tags didn't have an explicit order, etc. But overall it was not as painful as I expected.

I did write the XML document by hand. I didn't want to go through the hassle of trying to come up with a GUI to help me enter the information. Plus I have done enough HTML by hand that it didn't bother me. But if I do continue to improve this app I will give it a GUI as I don't recommend doing XML by hand when it can be avoided.

I made the decision from the start that I was not in the mood to use SAX for reading the XML document. If I was going to have to load the entire XML doc into memory I figured ElementTree was my best option. Biggest issue I had was making sure that when I read from the 'text' attribute on tags I needed to normalize the whitespace to get rid of extraneous newlines that were introduced from formatting the XML in a human-readable fashion. Otherwise ElementTree was very nice and easy to use. Thanks to /F for developing the package!

For templating I decided to go with Jinja. Georg Brandl's Python documentation redo uses it and so I knew it at least worked. Plus it uses Django templating syntax which is great since I was planning on learning it anyway. The hardest thing I had grasping was how template inheritance worked. I first I wanted my skeleton template to have various blocks that were defined in individual files that all extended the skeleton. That didn't work. Then I wanted my skeleton to extend the individual files like a mixin. That didn't work since multiple inheritance doesn't exist. I finally managed to think how I was supposed to think about template inheritance and ended up with a single skeleton that was extended by a single file that defined all of my code blocks. When I do more than one output format I will need to refactor some common things out into macros and other blocks.

It definitely helped that I was willing to refactor my XML schema while writing the templates. I quickly learned that using attributes as something for filtering on is not the best way to use XML. Ended up using grouping tags for several things that made my life easier. I also think there are a couple of other places that I could simplify the template code without sacrificing semantic information in the XML file that I might try.

In the end it was a pleasant experience. Overall the whole thing went smoothly and I would recommend any of the libraries I used to others.