Oplop 0.3 released

The password-generating program that I created, Oplop, has now hit version 0.3. The biggest change since the last version is that the whole idea of having various restriction filters placed on your eventual password is now gone. From practical experience it turned out that most sites are happy with what Oplop generates, otherwise sometimes there is demand for making sure there is a digit.

So now a digit is always included in any generated password. Simplifies the UI immensely. Now all one needs to do is input a label and a password to get their generated label password back; no more checkboxes!

And thanks to the power of App Engine, Oplop is online. I kept needing passwords at work and since I have not gotten around to writing an iPhone or Nokia S60 app, I decided I needed something. Well, since I am interning with the App Engine team and I already had Oplop implemented in Python, I figured I would just do an App engine application where I skip using Django and just do a bare-metal web app.

Now as of right now the online solution is not ideal. While I use HTTPS for everything to prevent online snooping, there is no protection from someone peering over your shoulder to notice the password that is shown on the screen. Besides, if I didn't have JavaScript so much or had my housemate mention he has had issues at his work with differing MD5 implementations, I could have a JS version that does everything client-side.

But I would like to harness the ability to store info on a per-user level for this web app. The thing holding me up is deciding how much to store. If I store labels along with any caveats for the label password for that label, then I don't have to think about having that stuff written down anywhere. Plus I could have completion suggestions for someone who is logged in. And if there is a way for someone to be logged into their Google account from an application then the label completion could extend to even application implementations of Oplop.

The security implication, though, is that if someone gets a hold of someone's label list, will that provide too much information? Chances are you could figure out where someone had an account from the label name (e.g. it's obvious what account the "amazon.com" label is for), but the labels are worthless without the master password. But if you got a hold of the master password, your accounts could all be compromised (still requires knowing the username for each account).

If I store just a list of sites where one has labels along with any password caveats for the site, then I could have a bookmarklet that tells you what the caveat is. You then still have to remember your label for the site, but at least that bit of information would not be available to anyone who manages to get a hold of your account information. But it does guarantee people will know where you have an account.

Storing both the label and corresponding site for the label gives you the perks of both, but with the combined drawbacks. I have not decided if any of these potential security issues are worth the benefits they bring at the moment.

Python == everything in the medicine cabinet

At least according to xkcd. And make sure to see the alt text for the image (usually just let your mouse hover).

pep362 module now up on PyPI

I just registered and uploaded pep362, version 0.4 on to the Cheeseshop (aka PyPI). The version number is so low because there are six open issues in PEP 362 that could potentially change the code. I am going to bump up the version number by 0.1 for each question I close.

I am going to guess that the module is the first thing uploaded to the Cheeseshop that works with both Python 2.x (my tests say 2.4 and higher because of generator expression usage) and Python 3.0 (at least a1) without any code modifications. If you look at the code it really was not hard to do. I actually had to add code to be compatible with 2.4 and 2.5 instead of 3.0. The real trick is if you like to toss print statements for debugging, you need to remember that in 3.0 print is now a function. Otherwise I have some 2.x and 3.0 specific tests but that is the only difference.

If you happen to use the code, please give me any feedback you have. That way it can possibly be worked back into the code and PEP to make this thing useful to others.

This means that item 1 on my latest todo list is finished. Item 2 I am actually going to skip as I don't think packaging up importlib is good until someone has tested it on Windows (if anyone gives it a go, please let me know).

ASCII mandelbrot in Python

I was reading Miguel de Icaza's (of Mono fame) blog entry about Luke Hoban's ray tracer one-line in C# 3.0 and LINQ. As Miguel said in the post, "I write the factorial function, Luke writes Ray Tracers". Rather cool.

So I checked out Luke's blog. The latest entry has some code in F# to generate an ASCII mandelbrot. Another cool example, I thought. Shouldn't be hard to reproduce in Python, right?

class ExhaustedException(Exception):
 pass

max_iterations = 100

def mod_squared(num):
 return num.real * num.real + num.imag * num.imag

def mandelbrot(outer):
 def mandelbrot_inner(inner, iterations=0):
     if mod_squared(inner) >= 4.0:
         return iterations
     elif iterations == max_iterations:
         raise ExhaustedException
     else:
         return mandelbrot_inner(inner * inner + outer, iterations+1)
 return mandelbrot_inner(outer)
y = -1.0
while y <= 1.0:
    x = -2.0
    while x <= 1.0:
        try:
            mandelbrot(complex(x, y))
            print " ",
        except ExhaustedException:
            print "#",
        x += 0.05
    else:
        y += 0.1
        print

Leads to the output of:

                                                                        # # # #                                        
                                                                        # # # #                                        
                                                            #     # # # # # # # # # #                                  
                                                            # # # # # # # # # # # # # # # # #                          
                                                        # # # # # # # # # # # # # # # # # # #                          
                                                      # # # # # # # # # # # # # # # # # # # # #                        
                                  # # # # # # #     # # # # # # # # # # # # # # # # # # # # # #                        
                                # # # # # # # # #   # # # # # # # # # # # # # # # # # # # # # #                        
          #         # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #                            
                                # # # # # # # # #   # # # # # # # # # # # # # # # # # # # # # #                        
                                  # # # # # # #     # # # # # # # # # # # # # # # # # # # # # #                        
                                                      # # # # # # # # # # # # # # # # # # # # #                        
                                                        # # # # # # # # # # # # # # # # # # #                          
                                                            # # # # # # # # # # # # # # # # #                          
                                                            #     # # # # # # # # # #                                  
                                                                        # # # #                                        
                                                                        # # # #                                        
                                                                                                                       
                                                                                                                       

(Sorry about Blogger chopping this off; just run the code to see the full output)

You might notice that compared to Luke's output I am missing the floating # at the very top and very bottom. I don't know why; probably some stupid error on my part.

Anyway, the biggest differences between the two is how the loops work and breaking out of the mandelbrot function. Oh, and that complex numbers are built into Python. =) Otherwise I didn't try very hard at all to optimize it, make it more Pythonic, or shorter.

There are a TON of top-level names in the stdlib

I just created a spreadsheet that lists every top-level name in the stdlib. How many rows do you think there are?

Give up? 272 entries. Out of those there are only 11 packages. That is a ton of code to maintain by only 42 people who have committed on Python in the past year.

Now granted, this is culled from the module index and thus includes __future__ and __main__, but still! That number is huge! I really hope we can trim the fat off the stdlib and stick stuff into reasonable packages.

Giving a talk on Python 3.0a1 in Vancouver on September 4th

On the evening of September 4th I will be giving a presentation on Python 3.0a1 in Vancouver, BC for VanPyZ (see the "Next Talk" on the VanPyZ site for details). We sometimes go for drinks afterwards and just talk. Group has always been really nice and it should be fun!

-Brett

General update

For some reason I just felt like writing and nothing miraculous has happened in my personal life, which mean I will be scratching my writing itch here on my coding blog. Sorry PythonLand. =)

So Python 3.0a1 gets closer and closer. Lately Guido has been trying to stomp failing tests with various people. I personally fixed test_warnings to be re-entrant (__warningregistry__ being set on individual modules is evil!) today. Skip Montanaro (who has over 200 commits on me according to Ohloh) has been removing support for old OSs. I believe Eric Smith is the one implementing PEP 3101 (the new string formatting implementation). And with the Google sprint coming next week (which I am missing; first sprint I have missed since I joined python-dev) we might actually hit alpha before the month is out!

As for me, I have mostly been hacking on importlib. Once I got the API redesigned to make it easier to have different storage mechanisms for source code I felt hyper-motivated to code. Pretty much all week I have been re-implementing something like an importer or loader and then writing the unit tests the next night. At this point I have rewritten the extension and source file importers with tests. The extension module loader has been rewritten and the basic loading functionality is tested, but I just tossed in PEP 302 optional extension support and that is not tested. The source code loader has been rewritten but the methods needs for the source handler has not been implemented so it is not functional.

But boy has this rewrite simplified the code! I must have been on a slight functional programming kick when I first designed importlib as I was passing around a lot of objects throughout out the execution flow and then querying them for stuff. Now I have a base class and then provide the methods and attributes the base class needs to work. Much cleaner and easier to follow now.

I can't knock functional programming too much, though, as I am still passing key objects around. But I have just thought out more exactly what information can be provided up-front and what needs to be conditionally asked later. That has led to smaller APIs to implement with the trade-off of slightly large method parameters. But since the info is needed anyway, making the info parameters really is not a bad thing.

Overall I am very happy with how the rewrite is going. Granted I should have branched off my import_in_py directory in Python's sandbox, but it's not a big deal as I am just leaving the code in-place that I am making obsolete until I can do a complete switch-over. The real test in all of this, though, is how easy/hard rewriting zipimport is using the new API I came up with. If that turns out to not be overly painful then I think everyone should be happy with the basic support the package will provide for writing custom back-ends that use Python source/bytecode (I have stopped worrying about DSLs based on Python for now as I don't want to over-design and it can still be handled as the code that handles the source/bytecode dance is just a function that is set on the source loader).

My first (real) experience with XML (through ElementTree) and Django Templates (through Jinja)

After asking people for advice about how to handle open source work in one's curriculum vitae, I wrote an app to help me manage my CV. I decided to use XML as the data format and then transform it somehow from XML into various other formats (text, HTML, PDF through LaTeX). Nothing fancy, although it was a first for me in terms of serious XML use from my own schema design.

To start, I wrote a rough XML schema using RELAX NG Compact syntax. This was not too bad, although I did have a battle myself over how often to use attributes compared to a full-blown tag. I also cheated in some spots and didn't properly specify when tags didn't have an explicit order, etc. But overall it was not as painful as I expected.

I did write the XML document by hand. I didn't want to go through the hassle of trying to come up with a GUI to help me enter the information. Plus I have done enough HTML by hand that it didn't bother me. But if I do continue to improve this app I will give it a GUI as I don't recommend doing XML by hand when it can be avoided.

I made the decision from the start that I was not in the mood to use SAX for reading the XML document. If I was going to have to load the entire XML doc into memory I figured ElementTree was my best option. Biggest issue I had was making sure that when I read from the 'text' attribute on tags I needed to normalize the whitespace to get rid of extraneous newlines that were introduced from formatting the XML in a human-readable fashion. Otherwise ElementTree was very nice and easy to use. Thanks to /F for developing the package!

For templating I decided to go with Jinja. Georg Brandl's Python documentation redo uses it and so I knew it at least worked. Plus it uses Django templating syntax which is great since I was planning on learning it anyway. The hardest thing I had grasping was how template inheritance worked. I first I wanted my skeleton template to have various blocks that were defined in individual files that all extended the skeleton. That didn't work. Then I wanted my skeleton to extend the individual files like a mixin. That didn't work since multiple inheritance doesn't exist. I finally managed to think how I was supposed to think about template inheritance and ended up with a single skeleton that was extended by a single file that defined all of my code blocks. When I do more than one output format I will need to refactor some common things out into macros and other blocks.

It definitely helped that I was willing to refactor my XML schema while writing the templates. I quickly learned that using attributes as something for filtering on is not the best way to use XML. Ended up using grouping tags for several things that made my life easier. I also think there are a couple of other places that I could simplify the template code without sacrificing semantic information in the XML file that I might try.

In the end it was a pleasant experience. Overall the whole thing went smoothly and I would recommend any of the libraries I used to others.

Curses is not fun

Like anyone tech-savvy person who is online, I have a ton of accounts. Obviously I should use a unique, secure password per account. But like most people that just doesn't happen as I can't remember that much. I could use a password management application, but it has not worked for me in the past as it requires me to have access to my computer in order to retrieve the passwords.

And then I came across an idea for site passwords by Alan Karp. Ka-Ping Yee used this idea for his Passpet add-on for Firefox. I really like the idea of using a single, very secure password and an account name hashed together (and passed through URL-safe base64) to generate a unique password as they look random.

Plus the portability problem is solved thanks to my cell phone. Since I can install Python on my phone I can write an app that implements the hashing algorithm. That way I have a way to get any of my passwords as long as I have my cell phone on me (which is pretty much all the time).

I had been planning on writing such an app to use on my laptop for several months now. While I was planning I realized that a GUI version would be nice. I also realized that because the application is extremely simple (two input fields, a way to specify to generate the password, and then displaying the password) I could implement it in various GUI libraries without having to learn all of the intricacies of the library.

And that is what I have done. I currently have a terminal, curses, and Tkinter version of the app. The terminal version was obviously simple; getpass and raw_input make it a no-brainer. Tkinter was surprising easy to use once I found Effbot's "An Introduction to Tkinter"; not pretty but the thing works.

But the curses version is just nasty! I had to piece stuff together from example apps, the docs, and random other places. I definitely consider my curiosity about curses satisfied. I understand the need it filled, but that doesn't make its use any less painful.