Mere Code

Black out

2010-01-26T22:13:00.002+11:00

I've been a bit slack and have yet to black out my website or avatar to protest the Australian government's mandatory Internet censorship policy.

In case circumstances prevent me from doing so, please read the Internet Blackout Australia website to see why it's so important. You might also want to read Hugh Blemings' well-written thoughts on the matter.

Electronic book reader?

2010-01-25T18:16:00.001+11:00

On my other blog, I ask about electronic book readers. Recommendations greatly desired.

Python testing goodies

2010-01-05T11:07:00.003+11:00

Just found out that there's a PPA with the latest releases of a whole bunch of Python testing goodies, including:

testtools - Extensions to unittest that make real extensions possible
testresources - Safely re-use expensive resources in tests without paying massive set-up costs
subunit - Manipulate, exchange and analyze test results without writing code
testscenarios - Run the same tests against many different implementations

If you are running Ubuntu 9.10 or later, then sudo add-apt-repository ppa:subunit will add it to your apt sources.

We're starting to grow a lot of useful, small testing tools that become even more useful when combined. I really like the "small pieces, loosely joined" approach, but sometimes that can make deployment & dependency management a colossal pain. Happily, Ubuntu, Rob and Launchpad PPAs to the rescue.

testtools 0.9.2 released

2009-12-16T09:55:00.002+11:00

I've just released testtools 0.9.2, which I firmly believe is the best testtools release ever. Thanks very much to Robert Collins and Benjamin Peterson for making it so.

For the last few releases, we've been working on being more than just a simple aggregation of existing unit testing best practices and tried to do our own experimental extensions.

These extensions aren't actually all that exciting by themselves. We haven't added better logging support or new types of outcomes or test replay or smart rendering of error results or anything like that. I'm pretty sure testtools will never do those sorts of things.

Rather, the extensions we've added are designed to let you do that, and then share your extensions with other people without getting them into the standard library's base unittest classes.

If you are using testtools, you can change the way TestCase.run() works without overriding run and without figuring out how to safely call user code. You can handle exceptions raised from tests however you'd like — again not needing to change TestCase.run(). You can add new, rich types of assertions without having to modify some base class somewhere. You can store information on a test object that can be used by a sufficiently smart TestResult, which can be handy if you want to see, say, access logs for all failed tests.

Of course, all this starts to get really powerful when testtools is in the standard library, and all of the other major Python test frameworks inherit from it: nose, py.test, zope.testing and Twisted Trial.

Even so, it's worth switching to testtools today, just for the assertion logic alone. All it takes is changing the base class of your test cases to testtools.TestCase. If your test framework supports running standard Python unit tests, it'll support testtools.

Lean Lean

2009-11-28T03:50:00.003+11:00

Ever been confused by people saying "this is Lean" or "that is Lean" or "we should actually be using a cyclometric value-stream set-based burndown kanban for this"? I have, so I thought I'd write a quick pared-down cheat sheet to explain exactly what Lean is. A lean version of Lean, if you will.

At the very least, Lean is seven principles which, we're told, we should apply to making software.

Eliminate waste
Amplify learning
Decide as late as possible
Deliver as fast as possible
Empower the team
Build integrity in
See the whole

The first principle is fundamental.

List taken from Lean Software Development.

How to ask an open source project to do something

2009-11-27T20:12:00.003+11:00

This isn't exhaustive, but I think there are two ways to ask an open source project to do something:

I want X. You should do X.
I want X. How can I help do X?

I have no data, but I think that asking the second way is what makes open source actually work.

To do: To dos (Palm Prē)

2009-11-26T20:22:00.002+11:00

You might already know that I have a lot of lists on my computer. I may have mentioned once or twice that I have such lists. These lists are connected to a certain popular productivity system which isn't really about productivity and is a little bit like a cult.

I might have also mentioned that I've been longing for a "smart" phone, partly so I can have these lists with me at all times, to clutch at reflexively like some Popish talisman.

Now that I have a Palm Prē I have lists and a smart phone, and yet one does not know the other. I am at a loss as to how to get my lists — line-separated text files on my laptop — onto the Tasks application on the Palm. Even then, once they are on the Palm, I have no idea as to how to get them synchronized with something I can edit from my laptop.

Any ideas?

Palm Prē reflections

2009-11-24T23:31:00.002+11:00

I got a Palm Prē yesterday. Here are some initial thoughts.

I like the feel of the physical device
The keyboard is small
Camera is useless for taking photos of receipts
The gesture UI is great
The Mail application shows individual emails, rather than GMail-style conversations. Lame.
I don't know how to get my lists out of text files on my computer and onto the Palm
I don't know actually know anything about syncing from the Palm
I still don't regret not getting an iPhone

UDS

2009-11-23T22:48:00.002+11:00

I'm back in London after a mammoth two weeks travelling the globe bringing Launchpad joy to its four corners.

First up I went to Mooloolaba, Australia. If you're Australian, you probably know how to pronounce it. If you aren't, you'll probably never get it. Sorry.

I was there with the Canonical Bazaar team and some other Launchpadders, talking about their next few months of development. They are going to be focusing on helping Launchpad and the Ubuntu platform team work on our daily build initiative. Hmm. I really need a place to link to for that last one, don't I?

Mooloolaba is a beautiful place to be in November. If you are thinking of giving me something for Christmas, consider a holiday apartment there.

After Mooloolaba, I crossed the Pacific to reach Dallas, home of the twenty-eight lane highway. I didn't really get into the city that much, but man, UDS was great.

Most of the people there are directly involved in Ubuntu development. It's such a joy to see so many people from all over the world come together to work on Ubuntu, and an honour to think that I have a place in that work.

I spent most of the conference flitting from room to room, going to sessions about the Ubuntu development process, daily builds and the Software Centre. Occasionally, I cheated a bit and went to a random session that interested me. Mostly it was about Quickly.

As a Launchpad developer, one of the most exciting things about this UDS was seeing that Ubuntu developers have started building their own tools around Launchpad using our API. Many of these tools are things we'd love to integrate with the main web app, but many are things that are so Ubuntu policy specific that we're happy to leave them outside. After all, this is exactly

what the API is about.

I did a short plenary session where I hacked up an API script using launchpadlib and demonstrated it live. After the presentation, many people came up to me and started talking about their API-using programs and their experience making them. The amount of interest really surprised me.

I also ran a couple of workshop / discussions. The first was on improving Launchpad for API use, which was a bit disorganized, since I didn't realize I was supposed to be running it. The second was on getting started with Launchpad hacking.

For the "Getting started with Launchpad hacking" session, I passed around a USB stick, and asked the audience for an API within Launchpad to expose. mdz suggested "blueprints", so I went to expose the 'drafter' element of a specification. In front of a room full of people. Sheesh.

If you think pair programming is hard (it is, and if you think it isn't you are either doing it wrong or you have extremely helpful colleagues), programming in front of a room full of people is harder.

Still, we managed to actually expose the API and propose the branch for merging with only a little bit of evil.

The point of the session was to get people set up with Launchpad development environments, to show them how to even begin to expose a method over the API but even more than that, to give people a little push into just giving Launchpad development a try. By blundering my way into a valid patch, I think maybe we succeeded.

Now UDS is over and I'm back in London, jetlagged and exhausted to be sure, but genuinely happy about the plans we have for Launchpad development, and eager to begin to make them happen.

Evil Overlord talks

2009-11-14T00:26:00.003+11:00

My "Hack Like an Evil Overlord" talks are now linked from my stuff page.

Tests: Costs and Benefits

2009-11-11T18:57:00.002+11:00

I like unit testing a lot. I've recently been thinking about whether it really is everything that advocates like myself make it out to be. I'm still thinking, but I reckon that language has something to do with it, and that I wouldn't like TDD so much if I were writing in Haskell. Maybe.

Anyway, here are the benefits of unit testing, without justification or explanation:

Easier to maintain code
Avoiding regressions
Make API clearer, both by describing it and by encouraging contracts.
Confidence in changes to code

For new contributors, there are other benefits:

Confidence in changes to code
Confidence in the validity of your own patch
Mechanism for exploring internal system behaviour

Some of the costs are:

Maintaining tests
Caring about the performance of tests
Running the tests
Debugging tests

For new contributors, there are other costs:

Finding tests
Figuring out how to run tests
Figuring out how to write tests
Learn unit testing

Is this fair? Is this clear? And exactly where was that post by that guy about unit testing being overrated?

On a half-related and slightly exasperated note, many of the conversations I've been having recently have been about costs and benefits. What happened to doing something because it was the right thing?

Raise your standards

2009-10-28T03:46:00.002+11:00

I gatecrashed LugRadio Live last Saturday and went to an excellent talk by Matthew Paul Thomas about reporting usability problems. Matthew has reported well over a thousand bugs against Launchpad itself (many of which have been fixed) so he knows something about the subject.

Matthew gave six pointers on how to report usability problems, but for me the single most striking one was this:

Raise your standards

When you are using an application with the aim of improving its usability, you need to shed all of your tolerance and forget those many workarounds you've learned. Drum your fingers impatiently while you wait for something to load. Think of how this would look if you were showing it to a sceptical friend.

The point struck me because so many times as an application developer I've thought to myself "it's good enough" or "it will work for now". Really though, that's not good enough. I should work until the thing I've added is invisible, or at least fun to use. I need to raise my standards.

Having raised one's standards, the trick is then to avoid the traps of perfectionism. Do you have any thoughts on how to walk the road, avoiding the pitfalls, standards held high?

Bug squashing sprints

2009-10-20T22:50:00.002+11:00

Every time somebody asks me what I think we should do for a sprint, I try to suggest "fix as many bugs as possible".

Each of the Launchpad teams gets together in a single location for a week long "sprint" fairly regularly. Maybe each team has two sprints a year.

Often these sprints end up being long talking sessions where designs for features are thrashed out. That's great and all, and sprints are a rare opportunity to do that, but often the design work goes unused for months while the team deals with the work already on its collective plate.

Sometimes these sprints have the whole team working (i.e. coding) toward a single milestone, be it a release or a major feature. These sprints are fun, but they've been quite rare for Launchpad. (Bazaar had a great one in Brisbane earlier this year).

What I'd love to see is a sprint where people fixed as many bugs as possible. Perhaps as a product strategist, I ought to be advocating higher-level visions and responding to the market and so forth, rather than saying "fix lots of bugs". But I don't think so, at least, not yet.

Fixing as many bugs as we can in a fixed time frame will make many users happy, since behind each bug is a user in pain. It will make us happy as a team, since a vastly dropped bug count will make us feel less overwhelmed and will feel like a concrete achievement. It's an easy sprint to measure the success of, and my hunch is that it would be a fun sprint too. It's also substantially different to what we normally do, which adds to the fun.

What do you think? I'd love to hear from people in community-driven open source projects, as well as people within Canonical who aren't in Launchpad. Maybe we're missing something that others know about.

Command line apps

2009-10-15T21:42:00.003+11:00

Back in the day, I used to want to use twisted.python.usage for all of my command-line apps. It's a fairly nice API & a good way of writing code.

Since I started writing Bazaar plugins though, I've fallen in love with Bazaar's command interface. Jamu has too, so he wrote Commandant. I don't really understand it though, which is mostly for lack of trying.

Michael turned our internal EC2-using testing tool into a bzr-a-like app using the Bazaar APIs. There's a bit of duplication, but it works pretty well.

The Bazaar developers are aware that their command-line interface code rocks and that it's too closely bound to Bazaar. I'm not sure whether anyone has filed bugs about it or not, but here's what I actually want:

The subcommand interface, e.g. "bzr foo"
Option parsing
Help
Command aliasing
Error handling
Progress display

There's also a bunch of cool stuff in Bazaar that's useful for a lot of applications, command-line or not.

Registries
Transports
Hooks
Oh yeah, plugins. (How could I have forgotten this?)

Sometimes I wish projects like Bazaar and Twisted split this sort of stuff out into separate packages. That would probably change the way the packages are maintained, and I don't know whether it would be for better or for worse.

Sadly, no moral or action for this post. Just stuff that's been on my mind that I wanted to write down and publish. I'd very much welcome input.

Launchpad extensions

2009-10-15T20:16:00.002+11:00

Launchpad has a pretty awesome public API, implemented using lazr.restful. I've written a few small scripts for it, and the Launchpad team has a few scripts that they use internally for doing admin tasks.

The Ubuntu Platform team does a heap of stuff with the Launchpad API. James Westby has been using it to make sure that there's a branch on Launchpad for every single package in Ubuntu.

There's all this great work, but there's been nothing to tie the room together. I've seen hardly any discussion about how to write Launchpad API applications, or how to test them, or how to get launchpadlib working in GTK+. I haven't even seen much code sharing.

So, borrowing a trick from Twisted's tx super-project, I've created an 'lpx' project group on Launchpad. Bring it your scripts, your applications, your huddled masses. If you want to know more about the API, look at the API help page.

Thanks to Mark for reminding me that this is important.

Launchpad status now on identi.ca

2009-10-09T20:06:00.002+11:00

Sometimes, sadly, Launchpad doesn't behave as well as we'd like.

Although we try extremely hard to always be available, and take each instance of downtime (planned or unplanned) as a personal insult, sometimes we're forced to disappoint both our users and ourselves. It's no excuse, of course, but at least we're in good company.

To make these inevitable interruptions easier to swallow, we've set up identi.ca and Twitter accounts that report the Launchpad server statuses, and only the server statuses. We've got people all around the world who are able to update it when things go bump in the night. There's also a link to the identi.ca page in our footer, so you don't have to keep consulting a blog.

Thanks very much to Matt Revell for getting this done.

Meta-meeting stuff

2009-10-09T05:19:00.002+11:00

Last week the Launchpad team leads gathered in London and had an absolutely huge meeting. Fifteen or sixteen smart, opinionated & passionate people in a room talking about the next six months of Launchpad development. It was a lot of fun.

When planning the agenda, Martin Albisetti and I were a little worried about things wandering off track, so we thrashed out a lists of dos & don'ts. I'll blurt them out here, with the really terrible ideas filtered out so I look more clever, and then offer some thoughts on how they worked.

Do

be happy with the results of this meeting.
ask questions, especially "why"
be punctual
stay on point

Don't

design features
check your email, i.e. laptops closed
leave the meeting without being clear
leave the meeting without being happy

The "laptops closed" one didn't work so well. It was great for the first couple of days, but by Thursday afternoon there were always people in a corner with those open LCD screens sucking chi out of the room like little dragons of despair.

The "Don't design features" rule was awesome. The rationale is that feature design leads to long discussions, and that it does a disservice to the many excellent engineers we have who aren't at the meeting and will be actually implementing a feature. At best, it's waste and at worst it's us telling people how to do something that they know how to do better. The clearest sign that it was a good idea came when other people started using it to shut me up.

Talking Time

2009-10-08T02:44:00.004+11:00

I've been working on the Launchpad team for a while, with most of that time being in Australia. Others in the team are in the US (red states & blue states), the UK, Germany, Brazil, Canada, New Zealand, Serbia, Lithuania and Thailand.

Here are some tips I've picked up for smoother online conversations, particularly around scheduling.

Don't say "summer" or "fall". Say the month, or the range of months.
Say "my morning" rather than "the morning".
Use 24-hour time.
In real-time conversations, say "in 2 hours time" rather than "at 10".
Always include the timezone. Avoid using local abbreviations (e.g. PST), instead use offset from UTC.
Better still, just give the time in UTC.
Know your UTC offset.
Use a timezone-aware meeting planner. You'll get the arithmetic wrong otherwise.
Say "Oct 7" rather than 10/7 or 7/10. Everyone speaks English, but not everyone uses your dialect.
The time that you end a meeting is more important than when you start it. Thoughtfully consider the timezones of other attendees when you are planning for & participating in meetings.

Violating these rules isn't a big deal, since people can generally figure out what you mean. Following them, however, can speed things along and sometimes even avoid tedious conversations.

Colon full of cookie

2009-09-15T16:16:00.002+10:00

Launchpad does translations. You can register a project on Launchpad and use Launchpad to translate your software into a zillion other languages. We've even got funky tools to take the translations that people do on the web and commit them straight to your branch.

As an ignorant Anglophonic monoglot, I've never really looked into Launchpad Translations. Software is already written in my language, and I can't help translate it into my language. There has been no reason for me to look at it... until today.

Barry Warsaw, hacker extraordinaire, has started a project called I have a colon full of cookie. This project aims to translate the phrase "I have a colon full of cookie" into as many languages as possible. Note here that "colon" is the organ, not the punctuation mark.

OK, time to bust out my Latin lexicon.

Layers are terrible

2009-09-14T18:49:00.003+10:00

When I talk about testing frameworks, I often mention Zope layers and say they are terrible. Some people have asked me for details on their terror and for justification of my opinion.

Here's all I've got. It's based on my experiences using layers with Zope 3.2 in Launchpad.

1. Layers have unnecessary magic

A layer can subclass another layer, which is fine. Subclassing means that your new, derived layer rests on top of the old, base layer.

When you define a derived layer though, you don't call the base layer's methods yourself, like you would for any other Python subclass. Why not? Because zope.testing magically up-calls the base layer's methods for you.

Yuck.

The upshot is that you write non-standard Python that confuses people not familiar with zope.testing. Also, you have no way of not calling the base methods, which is unfairly restrictive.

2. Layers are combined through inheritance

Each unit test is in a single layer. If you want a unit test to have the set up and tear down provided by layer A and the set up and tear down of layer B, you have to define a wholly new layer C that subclasses both A and B.

The layer C doesn't really mean anything other than "A and B". It doesn't really deserve a name, but it has to have one, since it has to be a new class.

Since layers are often used to share expensive resources between tests, you end up with a binary explosion of layer subclasses, as you move toward needing one for every subset of available resources.

This adds code and thus maintenance work, and is actually less clear than simply declaring that a test uses layers A and B.

3. Layers are implemented badly

Layers are all about changing the way a test is run. They supplement the run with extra work to establish a known base state for the test.

If you think about it, the stuff that's required to run a test is a property of the test.

Python's unittest module recognizes this. One of the few public methods of a TestCase object is its run() method. run() is also a method of TestSuite. This means that if you want to customize the way a test or a group of tests is run, you should override / re-implement the run() method.

Layers don't do this. Instead they change the way tests are gathered, the way they are reported and the way they are run. This means tests that rely on layers can only be run with the Zope test runner, and not with nose or trial or what have you.

4. Layers solve too many problems

Layers are not simply a way of sharing expensive resources between tests, they are also a way of using resources that cannot be torn down.

The implementation does this by detecting that a layer cannot be torn down, then spawning a new process to run the rest of the tests in.

It kind of sucks to have things that cannot be torn down in process, and it definitely sucks to conflate resource sharing with odd subprocess management.

5. Layers are not Zopish

As far as I can tell, one of the themes of Zope 3 is small, interchangeable pieces loosely joined together.

Layers seem to be something that could have been done much better using the Zope component architecture. Perhaps they could give getUtility some practical purpose in life.

What then?

As much as I hate to say so, I don't think there's much hope for layers as a concept. One is probably better served by replacing one's existing layers with testresources. It looks like it takes quite a lot of work to do so.

That's it. Five reasons and a lot of paragraphs. Please let me know if I've made mistakes, or if newer versions of zope.testing are better. Also, please gently correct me if I've left the path of civility and common courtesy.

Launchpad on Karmic!

2009-09-05T17:41:00.003+10:00

Thanks almost entirely to maxb & wgrant, I can now hack on Launchpad in Karmic.

This is great news for me, since it means I can ditch the chroot I was using. Check out the dev wiki for more details.

Active reviews, again

2009-09-04T16:57:00.002+10:00

You might remember I was really excited about the first version of the active reviews page, since it let me see at a glance the code reviews I needed to do, the ones I was waiting on, and reviews I could do if I felt like it. At the time, this was really important, since it's the first time I've seen a code collaboration tool orient itself towards me actually doing things. It's certainly a first for Launchpad.

Tim has recently landed a few changes to the page that aren't quite as revolutionary, but follow the same lines of thinking.

The one I want to mention is putting all the branches that are approved for landing onto the active reviews page. Every branch that's approved to land and hasn't yet landed is actually waste, it's an improvement that's sitting around gathering dust and waiting for someone to push a button. It's like a winning lottery ticket stuck to your fridge because you're too lazy to cash it in. We think it's so important that we've put these at the top of the list.

You can take a look at the active reviews pages for:

Now if you'll excuse me, I have branches to land.

Slow tests

2009-09-01T18:27:00.002+10:00

Robert Collins of Bazaar, recently blogged about the cost of slow tests. I couldn't agree more.

Launchpad has a very slow test suite. It takes upwards of five hours to get a change into our "known good" branch. It's been a source of pain for the team for years, and now that Launchpad is Free Software, it's becoming a pain to new contributors.

We've tried a few things to deal with it, none of them great. First up, we've got a tool for running the test suite on EC2 instances. It's nice, but it's still too slow. I've written a "faster tests" spec outlining some of the options we have; but specs never got software written. Likewise, there are a bunch of bugs flagged as build-infrastructure, many of which address the slow test sped. We also have a rotating "Build Engineer" position within the team to address those bugs.

It's probably not enough though. From watching Rob's blog posts, and the emails to the Bazaar mailing list, I think that what Launchpad really needs is someone with a passion for solving the problem (i.e. making Launchpad hacking fun) who is willing to lead the way, and a full commitment from the rest of the team for getting the runtime down and keeping it there.

More on this later.

Twisted & Monads

2009-07-26T11:18:00.002+10:00

On Thursday I spoke at fp-syd about Twisted and monads. All I have is the slides. If you'd like, I can write up the paper.

Launchpad now fully open sourced

2009-07-21T15:47:00.002+10:00

Launchpad has been open sourced. That includes Soyuz and Codehosting.

I'm now a full-time free software hacker. Squee!

Thanks to Mark for making the call, and to Karl for being the flux capacitor of the whole operation, humming along as we've built up to 1.21 gigawatts of freedom power and burst right through to the future.