Twisted & Monads

On Thursday I spoke at fp-syd about Twisted and monads. All I have is the slides. If you'd like, I can write up the paper.

Launchpad now fully open sourced

Launchpad has been open sourced. That includes Soyuz and Codehosting.

I'm now a full-time free software hacker. Squee!

Thanks to Mark for making the call, and to Karl for being the flux capacitor of the whole operation, humming along as we've built up to 1.21 gigawatts of freedom power and burst right through to the future.

unittest: it ain't broke, let's fix it

I spoke at EuroPython 2009 in Birmingham about the unittest module in the Python standard library.

The central thesis was first proposed to me by Robert Collins, probably around 2003 or 2004. The idea is that unittest is actually surprisingly good and more extensible than you think. It was also Rob's idea to co-present this as a talk in Birmingham. Sadly, I was obliged to present it by myself.

I'd like to write up the talk more thoroughly, and am keen to publish brief posts on why layers are bad and why setUpClass is bad, but for now I'll just post the artifacts from EuroPython.

• Slides (ODP | PDF)
  
• Text version of slides
• Notes
• Audio (mirror)
  

What do you think?

Update: Audio now available.

Documenting a Python project

I've recently started contributing a little to GTG, a GTD-inspired todo list application that's hosted on Launchpad.

So far, I've made a few UI tweaks and have done a bit of infrastructural work. I'm still trying to figure out how the code fits together. Since it's such a small code base, I don't mind writing a few docstrings for it. And if I'm going to write docstrings, I might as well generate some API docs, right?

And so the journey begins.

Everyone knows that pydoctor is the best way to generate API docs for Python code. So I fetched it from Launchpad and ran it according to the instructions:

$ bzr branch lp:pydoctor
$ cd GTG
$ ../pydoctor/bin/pydoctor --add-package GTG --make-html

This failed, saying it couldn't find zope.interfaces. I know I have zope.interfaces on my system:

$ sagi python-zopeinterface
...
python-zopeinterface is already the newest version.

But, ah, this is karmic, and pydoctor is insisting on running with Python 2.4, which is completely broken on karmic. I tweaked the bin/pydoctor file to use the default Python and then ran the command again.

$ ~/src/pydoctor/trunk/bin/pydoctor --add-package GTG --make-html

Success, I think. There's a lot of output, so it's hard to tell. Trying again with a quiet option:

$ ~/src/pydoctor/trunk/bin/pydoctor --add-package GTG --make-html -q
WARNING: guessing GTG for project name
/home/jml/src/Divmod/trunk/Nevow/formless/annotate.py:730: DeprecationWarning: object.__new__() takes no parameters
rv = cls = InterfaceClass.__new__(cls, name, bases, dct)
/home/jml/src/Divmod/trunk/Nevow/nevow/testutil.py:7: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
from popen2 import Popen3
/home/jml/src/Divmod/trunk/Nevow/nevow/guard.py:15: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
these 6 objects' docstrings are not proper epytext:
GTG.core.tagstore.Tag.__init__
GTG.core.tagstore.Tag.set_attribute
GTG.core.tagstore.Tag.get_all_attributes
GTG.core.requester.Requester.new_task
GTG.core.requester.Requester.get_tasks_list
GTG.core.requester.Requester.get_active_tasks_list

Hmm. The WARNING sucks, but I can work around that by specifying the project name on the command line. The deprecation warnings also suck, but I'm not quite sure what to do about them. For now, I'll invoke python with '-W ignore::DeprecationWarning'.

This is as good a point as any to point out that I had Nevow trunk already in my PYTHONPATH. I don't know what version of Nevow you will need.

The 'not proper epytext' warnings are more worrisome. I thought I'd been writing proper docstrings. Also, it doesn't tell me what the actual problems are: I need to drop the '-q' option for that. This gets me too much information. Oh hello, there's a '--verbose-about' option. Digging into the source doesn't enlighten immediately, but bzr-grep eventually reveals '--verbose-about=epydoc2stan2'. If I repeat '-q' twice and the '--verbose-about' option twice, I get precisely the output I want.

python -W ignore::DeprecationWarning ~/src/pydoctor/trunk/bin/pydoctor \
--add-package GTG --make-html --project-name=GTG \
-q -q \
--verbose-about=epydoc2stan2 \
--verbose-about=epydoc2stan2

What I really want is a pyflakes-style checker for my docstrings that I can hook up to flymake. That way, I can be warned about my poorly-formed docs as I'm writing them. Looking at the pydoctor help, it doesn't look like such a thing exists. The command above is close, but it writes too much output to disk and takes far too long to run.

Also, it's not quite right. It seems I've been writing my docstrings in the style we use in Launchpad, rather than in epytext style. So if I use '--docformat=restructuredtext', my docstrings are formatted properly and I get fewer errors:

python -W ignore::DeprecationWarning ~/src/pydoctor/trunk/bin/pydoctor \
--add-package GTG --make-html --project-name=GTG \
--docformat=restructuredtext -q -q \
--verbose-about=epydoc2stan2 --verbose-about=epydoc2stan2

This leaves only two formatting errors, both valid. It takes a full second longer to generate the docs, which sucks. Still, we've got doc generation, and we've got valid errors.

The output is good, but it's not great. Take a look at the Twisted API docs, for example. I can't quite put my finger on it, but I wish that someone with a vision and the kind of CSS skills that you have to buy from the devil at a crossroads at midnight would set themselves the task of making the API docs look great.

So where does this leave us:

• I can generate pretty good API docs easily and get told only about interesting errors
• I need a pyflakes-like docstring formatting checker
• I need a simple docstring coverage checker. I do not want to make it part of my test suite or build process.
• I had to tweak the pydoctor source before being able to run it from trunk
  
• The output is OK, but it ought to be beautiful
  

testresources 0.2

At EuroPython this year, I talked about unittest and how it's misunderstood little module with it's own problems, and that despite its rough exterior it really is a Good Thing.

To demonstrate this, I gave examples of three pieces of software that each extended unittest to solve the same problem: sharing expensive resources between tests.

The first two examples were Twisted's setUpClass / tearDownClass and Zope's layers. They are terrible and you should never use them.

If you really want to share expensive resources between tests, you should use testresources. testresources makes it easy to write tests that use resources that cost a lot to set up and tear down (such as databases or SSH servers) without screwing up your test isolation. Robert Collins, the author, cut the 0.2 release today.

testresources now has exactly one known bug, although Rob argues that maybe it's one unknown bug. It's down to the quantum level here: that's how good testresources is.

New GPG key

I've published a new GPG key: 2048R/2A727997

Details available here: http://static.mumak.net/jml-key-transition-2009-07-13.txt

Angle on code review

You're a core contributor to a Free Software project. Maybe you get paid to do it, maybe you don't. You get a patch from the outside: someone whose name you don't recognize has contributed a patch.

You could ignore this patch, since you've got a lot of work to do already. Better, you could review the patch and give the newbie some much needed feedback.

Alternatively, you could champion the change by making sure it gets the treatment that one of your own patches would. Review it or get it reviewed, help the newbie make the changes the reviewer wants, make sure it's managed using the best systems for managing patches in your project, keep track of its progress like you would one of your own patches, be eager to see it landed, pimp it in the changelog and the release announcement, get the newbie's name in lights.

In practical terms: if you use a DVCS, get the patch out of whatever email attachment its in and put it in a branch; if you use a code review system like Launchpad, get it out of the bug tracker and into the code review tool; if you have a test suite, empower the newbie to write tests (normally that means knowing where to put them and how to run them).

It's a matter of perspective: am I here to maintain the quality of the code base and prevent bugs from being introduced, or am I here to help others in my community. Am I just a reviewer or am I a champion?

Twisted huh?

One of the hallway conversations I've had a few times so far at EuroPython goes like this:

A: Twisted's got some good stuff going for it, but I don't like the way it forces me to use Zope interfaces. It's too heavyweight?
B: Huh? What do you mean? It doesn't make you do that.
C: Yeah, there's lots of deep inheritance, which just feels really wrong.
A: Right, I have to write too much code to get something simple working.
B: Sorry, I have no idea what you mean. Can you give a concrete example?
A, C: I can't think of one right now.

To me, this is a very intriguing beginning to a real conversation. Do you know what A and C might mean here? I'd love to understand their points so that I can form an opinion.