Doctests

Andrew has written about why narrative tests make lousy unit tests and problems with the doctest format. In summary:

• Unit tests work better if each one has a name that identifies it.
• Specific, isolated tests give clearer failures, and are easier to debug.
• Specific, narrow tests are better at communicating intended behaviour.
• Comparing two objects in doctests is hard.
• It's hard to get an overview of what's tested in a particular doctest file.
• Doctest is a mini-language that's worse than Python. It's got corner-cases and outright bugs.
• Tests are code, and code works better in .py files than .txt files. In particular:
  • Python has better tool support. Syntax highlighting, code folding, pyflakes, 2to3 etc.
    
  • It's easier to build test infrastructure in Python.
  • Test code benefits from refactoring as much as regular code, but doctests make it hard to do this.

Of course, in the end, it comes down to this:

It is just as possible to write incomprehensible tests using doctest as it is using TestCase classes with test methods.

Review Thoughts

So, I think I've figured out what it is I dislike about Twisted's review process: reviews aren't thorough enough.

This sounds a little weird, since it's actually really hard to get a patch into Twisted: it almost always takes me at least three round trips just to get something in. But I think the number of round-trips is actually a symptom of this lack of completeness.

In Launchpad, reviews are done as in-line replies to diffs. A reviewer is obliged to note each chunk of code that needs to be changed, along with exactly what needs to be changed. In Twisted, reviews are done as Trac comments and generally provided as bullet points. In Launchpad, a reviewer would say, "You need to change foo_bar to fooBar, because our coding standards require camel case". In Twisted, a reviewer might say "There are some naming convention issues".

This obviously varies between reviewers and even between reviews, but I think that the difference in technologies encourages differences in review style.

As a patch submitter, I find the in-line-comments-on-diff form much more helpful. It provides me with a convenient todo list, and it lets me know that the reviewer has looked through and tried to understand all of my code. It essentially turns the review into a series of mini bug reports with "observed, expected, how to reproduce" sections (where "how to reproduce" is "where to find").

I also like it as a reviewer, since it means less typing.

Papers and Talks

At the Launchpad Epic, I presented a paper on Twisted for the other Launchpadders. I think that the paper itself is a good introduction to how Twisted works and why, and I'll be publishing it here once I get a chance to format it for the web.

I did two lightning talks that might be interesting to readers. One was on testresources and the other was on how to hack like an Evil Overlord. I think I'll try present the second one at OSDC.