Mere Code

Diverse Topics of General Interest to the Practicing Programmer

Deliberately being naughty

I don’t get a whole lot of time to hack on things outside of work. Maybe one or two slots of about two hours in a good week. What I want is for that time to be fun, creative, and hopefully useful for others.

When I do get such discretionary time, what I often end up doing is code review. I believe that finishing things is more important than starting them, and know how much it sucks to be waiting for someone to review your code.

So, I’ll do a couple of code reviews (some of which might be strictly blocking on me). And then I’ll run out of time.

This means I rarely work on anything I actually find fun. Code reviews are rarely fun.

There’s a similar, related thing working on a project that’s got a backlog of code reviews, like, say Twisted. Twisted has forty or fifty patches right now that are waiting for someone to review them. Do I want to fix bugs in Twisted? That would be contributing to the patch problem, so probably not. Do I want to do code reviews? Well, I guess I should, but I’ll probably find something else to do instead.

I don’t think Twisted should change its policy. I think the solution is for people like me to volunteer more time doing something they don’t necessarily enjoy for the sake of achieving some broader, nobler outcome.

However, for me to sustain myself and keep enjoying open source and my chosen career, I’ve decided to make sure that I regularly schedule “naughty time”. That’s going to be time where I don’t work on the thing that I should be working on, but instead play with something I find interesting at the time, even if it’s not the best thing to do.

testtools moved to Github

Those who closely follow testtools development will notice something unusual in the 0.9.28 release: we now host the code on Github rather than on Launchpad.

There are a few reasons behind the switch, but the biggest one is that the ecosystem of tools and services around Git & Github is much, much larger than that around Launchpad & Bazaar.  Already, by using Travis (thanks kampka!), I’ve been able to kill off an old, cruddy, insecure Jenkins installation on my server, and actually get proper testing across all of the Python versions that testtools supports.  No more, “oops, I broke 3.2” again.

We still track our bugs on Launchpad, because Launchpad’s bug tracker is still our favourite.

Happy hacking!

Are single letter variable names evil?

A colleague recently asked the question, “Are single letter variable names evil?”. He was trolling, and I kind of knew it, but gosh, what an excuse to rant about code clarity! My comments ended up being multiple pages long, so taking one of Mary Gardiner’s suggestions to heart (although I’m sure she regrets it now), I’ve turned it into a blog post.

In a nutshell, I don’t think single variable names are evil. Lots of the time they are a bad idea though.

The classic rule of thumb I’ve heard is that “the length of the variable name should be proportional to the log of the size of the scope”. I can’t find who said that originally. My best guess is Mike Haertel. It seems to be a good rule of thumb.

But the corollary is that sometimes when the code is confusing, the problem isn’t short variable names, it’s that the scope is too large.

Perhaps there’s also something to be said for shorter names being more useful for really generic code? I’m OK with an implementation of map taking either f and xs or function and things. I don’t think either is better or worse.

And maybe single letter variable names are acceptable when they are the standard terms for the problem domain. (x, y) might be clearer than (horizontal_position, vertical_position).

Finally, Python’s idiomatic filtering strongly encourages short names, because you have to repeat it three times within the scope of one expression:

[x for x in meaningfully_named_list_of_things if x.relevance > THRESHOLD]


 for meaningful_name in meaningfully_named_list_of_things
 if meaningful_name.relevance > THRESHOLD]

It’s probably subjective, but I think the second one is less clear, since it obscures what’s going on. Bice Dibley points out that in these list comprehensions, x functions much like a pronoun in English.

As is the case when writing prose, being clear is hard. Rules can help – taking knowledge and distilling it into formal principles is one of the best ways for us to progress as a species! – but as Orwell said:

Break any of these rules sooner than say anything outright barbarous.

George Orwell Politics and the English Language

(And asking someone to look over it soon will probably spare troubles later)

Getting set up with Twisted development

I haven’t contributed much to Twisted recently, but today I’d really like to get my new Deferred documentation ready for review.

To my knowledge, the only sensible way to actually work on Twisted code is using Combinator. With it, I can make branches, commit to those branches, merge in updates from trunk and when my branch is approved, actually land it. All of this is described more fully in Twisted’s pioneering Ultimate Quality Development System.

Combinator works by convention, so its source must live within a Divmod/trunk directory. Since I have the code in a colocated Bazaar branch in ~/src/, I created a new directory under ~/src/ called Divmod and symlinked trunk to it, approximating the instructions on the Combinator page.

  $ cd src
  $ mkdir Divmod
  $ cd Divmod
  $ ln -s ~/src/ trunk

Trying to actually use Combinator only got me stack traces and warnings though, so I poked around in the source code and now have a shell script in ~/src/ that does this:

  export COMBINATOR_PROJECTS=/home/jml/src
  eval `python ~/src/Divmod/trunk/Combinator/`

Sourcing that script works:

  $ .  ~/src/

After that, I blew away my checkout of Twisted and made a newer, Combinator-friendly one:

  $ rm -rf ~/src/Twisted
  $ chbranch Twisted trunk \

All a little bit of a hassle, but worth it to get back into the swing of contributing to Twisted.


Jonathan Lange on 2012-12-18 09:08
Thanks. This is largely written so I can refer to it later.
glyph on 2012-12-18 07:09
Thanks for posting the refresher. I hope we can make it less necessary for people to jump through this hoop in the future, but until we do that it's nice to have people making these ad-hoc updates to the docs!

What’s wrong with ORMs

I was recently reading about F1, Google’s fault-tolerant distributed RDBMS and came across this neat little summary of what’s wrong with ORMs.

  1. Obscured database operations
  2. Serial reads
  3. Implicit traversal
Obscured database operations essentially means that when scanning code, you cannot easily tell which code is going to kick off a network round-trip to the database. If you care about writing code that performs well, it’s pretty important to know.

Serial reads are what many in my circles know as ”potato programming”. When you have a loop, and something inside that loop is doing a query, that’s potato programming. Again, disastrous for performance.

Implicit traversal is doing unwanted joins and loading unnecessary data. ORMs tend to load up an object with all of its instance variables, which often means traversing the graph of references.

Anyway, this is a much handier answer to have than ‘google for ”Vietnam of Computer Science”’, which is fascinating but rather lengthy. (See also Jeff Atwood’s summary).

For bonus points, these three things are also what’s wrong with Launchpad’s API.


Jonathan Lange on 2012-11-21 10:10
For 1 for Launchpad's API, change "database" to "remote". It's really easy to accidentally do something with a launchpadlib object that makes it hit the network, giving your lucky, lucky users another roundtrip to wait on.
Aaron Bentley on 2012-11-21 04:46
I agree about 2 and 3 affecting Launchpad's API. For 1, don't all Launchpad API operations hit database? And so how is that obscure? Unless you mean Launchpadlib there.

A Modest Success

Over the last few months, James Westby and I have been working on automatic packaging. I want to talk today not about the thing itself but rather how we go about making it, because I think that we have got a fairly good process going.

Here’s what it looks like.


We pull the work that we do from a backlog of ideas and requests. For example, a little while ago, we did some experiments that showed that if we changed the way we mapped libraries to Ubuntu packages, we could get much better results for automatically packaging binary apps. The project to change this ended up in our backlog, and we recently decided to work on it.

Before we begin work, we like to have a definition of “done”. We also write down why we are doing this, and what our assumptions are. We then break things up into tasks. Maybe I’ll dig into this another time.


Most of what we do goes in a loop:

  1. Make a branch
  2. Work on it
  3. Propose for merging
  4. Code review
  5. Automated tests-then-land
  6. Automatically triggered staging deploy
  7. Manual “is it deployable” check
  8. Request production deployment from webops 
We use Bazaar for version control and we use Launchpad for hosting code and doing code review. Here’s an example one.

Code review is quite often fairly boring. My guess is that one in ten actually provoke interesting changes. Most of our branches make small changes, all of our changes are covered with tests, and we have confidence in our test suite and our ability & willingness to make follow-up improvements. Thus, rather than seeing code review as our last line of defence for ensuring code quality, it’s an opportunity to learn and to perhaps accelerate the process by spotting problems earlier.

Once a merge proposal is approved, magic happens.

In this case, “magic” is a Jenkins instance that uses tarmac to poll Launchpad to figure out which merge proposal are approved and thus ready to land.  It runs our test suite, and if it passes, lands the change. If it fails, it posts the failure to the merge proposal and our IRC channel and marks it as needing more work.

It’s important here that our test suite is fast, that it’s reliable, and that it gives useful error messages when things fail.

When the change is to a dependency, then we run the tests of the things that depend on it with the latest change.

After a successful change to trunk, we have a Jenkins job that triggers an automatic deployment to our staging server.

All of this takes less than five minutes.

Once it’s on staging, our “QA” website lists it as a revision that needs to be manually verified. What we do there isn’t really QA, but instead making sure that if we roll it out to production, we won’t break anything. As with code review, we are confident in our ability to fix things later.   

We tend to do these in batches, as our webops are insanely busy. Once we’ve got a group of changes that are safe to deploy to production, we ping the webops, who then (I think) run a single command that deploys to production very quickly. Sometimes because of interrupts, it can take twenty minutes to an hour to get someone to do this.


At this stage, we return to our definition of done and check to see if the change actually provides the value we thought. 

Because automatic packaging is an inherently fuzzy problem, we run a barrage of real world data through the new system to see if the recent changes actually help. This also generates interesting qualitative data that gives us hints on where we might want to work next.


This is the end result of a lot of work, mostly by James Westby with me cheerleading from the sidelines. We’ve iterated through it a lot, making things faster and more reliable, generally by going from polling to events, by cutting out unnecessary review steps, and by making visibility improvements for when things are broken.

Underlying it all are a few principles that we have found to be either true or useful:
  • a thing is not done until users are happily using it
  • more strongly, a thing is not valuable until users are happily using it, until then it is wasted effort 
  • we should be able deploy any change in the main code or a dependency within minutes at any time, every time
  • all deployments must be from trunk, all revisions of trunk must be tested and known-good
  • we can avoid a lot of needless effort by asking ourselves why we are doing what we doing, and by being explicit about our assumptions
  • regressions are intolerable, everything else is tolerable, because we can, will and do fix it
Rather than spend time shaping up this information into a pitch for why doing things our way will bring you fortune and glory, we just want to use this post to do the bare minimum of making what we are doing known more broadly.

Do you do something similar? Is there something above that you would like to know more about? Are we making claims that seem a bit far-fetched? Are any of our principles BS? Have you tried working toward goals like these?

We would really love to know and to hear from you. It will help us get better. Also, I’ve deliberately fudged some stuff, so it’d be great to get called out on that.

Thanks to James Westby for reviewing this post. All errors are mine.

Undistract me (redux)

In January, I told you all about a terminal hack to help me recover from the inevitable distraction that comes when I run commands that take a while to run (for me, mostly test suites).

I’ve been using the hack since then, and it has given me a great deal of pleasure. Others have watched it and said that they very much wanted it.

As such, you can now go to the undistract-me website to get instruction on how to download and install it. Or, if you’re using Ubuntu:
$ sudo add-apt-repository ppa:undistract-me-packagers/daily
$ sudo apt-get update
$ sudo apt-get install undistract-me
Once you’ve done that, and have started a new login shell, then you’ll get a notification whenever a command that takes more than 10 seconds to run finally completes.

Thanks to Glyph Lefkowitz for the preexec hack that made this possible, and to Chris Jones, Clint Byrum and Michael Vogt for their help in figuring out what I should do.


Jonathan Lange on 2012-10-04 16:06
Thanks for the bug! I've replied in detail there.
Andrew on 2012-10-04 14:41
undistract-me doesn't work for me in Ubuntu 12.10. I've filed a bug:

Rigor mortis?

I’ve been on a bit of a sanity bender recently: science, logic, evidence, experimentation, clarity and things like that. Here’s a short list of some of the things I’ve been reading:

If you’re writing software then I recommend “Pretotype It” first, because it’s likely to make you write less software, and that can only be a good thing. They are all great reads though.

There are a few general themes: be explicit about your assumptions and try to verify or falsify them as soon as you may; do experiments; beware of certain mistakes or logical short-cuts; learn statistics; understand what you are saying. Wonderful notions all, but I’m not sure whether they are working out for me.

Occasionally I’ll get an email that has a couple of sentences but somehow manages to squeeze in all sorts of conflations, non sequiturs, and general fudging. It’s hard to know where to begin. I can make a fair stab at analyzing the errors, but synthesizing a response that actually helps is very hard.

Rigour also puts a restraint on rhetoric. It’s hard to say something convincingly when you have a bunch of qualifiers dangling at the end. My writing (even now!) is slowed down as I notice the unfounded assertions and unstated assumptions that lie behind it.

Also, much of this doesn’t help you get from a vague, interesting intuition to a workable idea, from hunch to hypothesis, if you will. Sometimes a notion needs to time to grow before it’s rejected as irrational or incorrect. Something like Thinking Hats can help here.

This is all peanuts though. Do more science. Really.


glyph on 2012-09-02 00:56
I find that, when confronted with a muddled mess of ideas in email, friendly, direct communication is best. Especially because people will almost always surprise you with their capacity to improve their communications, and you'll rarely have to write a "these are the twenty-seven logical fallacies that you have committed in this message" kind of message twice.

It can take some practice to make this kind of communique not come off as super condescending, but it's definitely doable. The trick is to keep the focus on the communication and not the person or the ideas.
Jelmer on 2012-09-01 15:22
I just finished Pretotyping; that was an interesting and useful read. Thanks for the suggestion.

testtools: history and future

Before I started testtools I was switching regularly between bzr, Launchpad and Twisted and was sick of one excellent testing innovation being available in one and not the other. I tried submitting some of the improvements to Python. They were either ignored or rejected.

Also, as the maintainer of Twisted’s testing framework, Trial, I had been burned several times when the standard library would change its API in backwards compatible ways, so I wanted something that would encapsulate all of that once and for all.

So, to provide a temporary ground between big, upstream projects that are serious about testing and Python’s standard library, I made the project that was to become testtools. (It was originally called pyunit3k. Sorry.).

Its constraints were that it should work with Python 2.4, 2.5, 2.6, 2.7 and 3.1 (Bazaar, Launchpad & Twisted all cared about Python 2.4 at the time), that any code in it should be allowed to go into Python without special permission from contributors, and that it wasn’t to have any frivolous innovation, patches should be of code proven to be useful elsewhere.

Then eventually Michael Foord started maintaining Python’s standard unittest. While we haven’t always seen eye-to-eye on the details, it’s definitely been a huge improvement, and Michael deserves props for doing a job that I was unwilling to do. Releasing unittest2 was a great step, and I think a good model of how any standard library maintenance should be done.

Since then, we’ve lightened the restraint on innovation in testtools, and the last release was our last release to support Python 2.4 and 2.5. The people who use it really love it, we’ve had plenty of contributors, I use it on heaps of projects, and it certainly makes my job much, much easier.  Launchpad and Bazaar use it, but Twisted probably won’t ever, so that’s 66% success according to my original motivation. 

I wish more people would use testtools. Actually, no, that’s not right. I wish more people would use the things in testtools. If people do that, it makes it easier for me to contribute to their code, and makes using their testing innovations much easier. 

As such, I’m hoping to try to get some of the most useful bits (matchers first) into upstream Python. Wish me luck.

Complexity and capability

“You might think that some people are better at understanding complexity than others, but relative to the sheer scale of the complexity that we can all create, we are all about the same.”

I think Rich Hickey said this. It’s true.