Mere Code

Have you tried lptools?

2010-03-12T04:17:00.002+11:00

Have you tried lptools?

It's not at all officially associated with the Launchpad project, of course, but it's got a few nice things that you might want to look at, including:

a milestone manipulator
a code review notifier, and
a milestone to iCal exporter

One of these days, I'd like for there to be an official, awesome Launchpad command-line client. As it is, I'm happy with the world of people making extensions to meet their needs.

For other extensions, check out the lpx project or our list of clients.

launchpadlib powerup

2010-03-11T05:54:00.002+11:00

In my last post, I introduced launchpadlib and demonstrated a very simple script that uses it. In this post, I'd like to build on that a bit and show you how to do something actually interesting.

In particular, I want to show you how to search for bugs, teach you a bit about Launchpad's internal data model and help you help yourself when it comes to figuring out Launchpad APIs.

The script at lp:~jml/+junk/bugstats is designed to tell you how good you are at filing bugs. It uses a very simple metric: out of the bugs that you've filed, how many actually have been fixed.

$ ./bugstats.py ubuntu jml
jml is 22.22% successful on bugs in Ubuntu
$ ./bugstats.py launchpad-code jml
jml is 47.63% successful on bugs in Launchpad Code

To do that, we need to:

get the "project" and person referred to on the command line
search for all fixed bugs filed by that person
search for all bugs in total by that same person
count them both
divide them
print them!

I say "project", but I really should say "pillar", which is the Launchpad technical term for a project (e.g. "bzr"), distribution (e.g. "ubuntu") or project group (e.g. "gnome"). A pillar is anything in first part of Launchpad URL that isn't a person.

We get the pillar and person like this:

   pillar = launchpad.projects[pillar_name]
  reporter = launchpad.people[reporter_name]

Pretty easy, huh? Now, how do we search for bug tasks?

The first port of call is to go to the Launchpad API reference page. I'm going to look for the string 'reporter', since that's the one thing I definitely know I want to find.

Eventually, I found the searchTasks method (named operation) that's on pillars and takes a bug_reporter parameter and a status parameter. It returns a collection of bug_tasks, which are the objects that represent the rows in the table you see at the top of a bug page.

I can find the bugtasks for the bugs I've reported that have been fixed by doing:

   fixed_bugtasks = pillar.searchTasks(
      bug_reporter=reporter, status=['Fix Released'])

It took me a while to figure out exactly how to spell "Fix Released". I ended up using trial and error.

Similarly, I can all the bugtasks for bugs I've filed by doing:

   total_bugtasks = pillar.searchTasks(
      bug_reporter=reporter,
      status=[
          "New",
          "Incomplete",
          "Invalid",
          "Won't Fix",
          "Confirmed",
          "Triaged",
          "In Progress",
          "Fix Committed",
          'Fix Released'])

I cheated a bit for that one and looked at the launchpad code to get a list of all bug statusus. The default for searchTasks is to only return open bugs.

Once we've got the collections of bug tasks, we need to get their counts. In an ideal world, it would be len(total_bugtasks), but sadly bug 274074 means that len is really, really slow here.

Instead, I wrote this helper function:

def length(collection):
  # XXX: Workaround bug 274074. Thanks wgrant.
  return int(collection._wadl_resource.representation['total_size'])

With that, I can calculate & print my success rate at filing bugs:

   percentage = 100.0 * length(fixed_bugtasks) / length(total_bugtasks)
  print "%s is %.2f%% successful on bugs in %s" % (
      reporter.display_name, percentage, pillar.display_name)

Next up on the API, I'll talk about some of the gotchas and what you can do about them.

Get started with launchpadlib

2010-03-10T00:34:00.003+11:00

In my spare time, I sometimes talk to people about how they can get started with launchpadlib hacking.

launchpadlib is the Python client-side library that talks to Launchpad's own REST API. It turns out that customize scripted control of a bug-tracker-code-hosting-translation-distribution-building-cross-project-collaboration thing is actually quite handy.

If you want to get started hacking with launchpadlib, and you have Ubuntu, then install 'python-launchpadlib' now. I'm pretty sure you can also get it from PyPI.

You can check that it works by running:

$ python
>>> import launchpadlib
>>> launchpadlib.__version__
'1.5.1'

I'll be assuming you're running 1.5.1 or later.

I've written a very simple launchpadlib application that you can get with 'bzr branch lp:~jml/+junk/bugstats'. Each revision shows a meaningful launchpadlib script. You can get at the old revisions with 'bzr revert -r1' or 'bzr revert -r2' or '-r3'.

Here's what the simplest launchpadlib script that I could think of looks like:

import os
import sys
from launchpadlib.launchpad import Launchpad, STAGING_SERVICE_ROOT

APP_NAME = 'jml-bug-stats'
CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache')
SERVICE_ROOT = STAGING_SERVICE_ROOT

launchpad = Launchpad.login_with(APP_NAME, SERVICE_ROOT, CACHE_DIR)
print launchpad.bugs[1].title

(Adapted from r2 of the above branch).

A few points.

We use STAGING_SERVICE_ROOT, which means that we're pointing
at Launchpad's staging service,
just in case we screw up any data.

We give the application a name, when you run the application, launchpadlib
opens up a browser window letting you decide how far the application
can act on your behalf.
We provide a cache directory. Credentials, among other things, get stored
here.
We then login and get an object that represents a Launchpad instance
Once we've got it, we look at the collection of bugs, get Bug #1 and then
print the title

Very simple. To learn how to write this application, I looked at the main Launchpad API help page, the
examples page and the reference documentation. You'll notice that I had to translate the reference documentation from REST-speak into Python-speak.

Already you have enough to go exploring with the Launchpad API and think of cool things to do. A bunch of people are already doing cool stuff and there are many projects that use launchpadlib.

Next up, I hope to show you some more complex things you can do with the API.

Monitor & keyboard

2010-03-08T23:25:00.002+11:00

I want to get a shiny new monitor and keyboard. The monitor is for coding, writing docs, answering emails and watching films. The keyboard is for same. They'll both plug into my Thinkpad X200 for now.

I'm tempted to get an Apple Cinema Display (but at such cost!). I don't know what keyboard I want. I think that the DAS keyboard I have isn't it.

Thoughts?

Back from PyCon

2010-03-03T20:54:00.002+11:00

I had a great time at PyCon 2010. The best part was definitely the Twisted sprint, which gave me a long overdue opportunity to do some actual coding.

Well, actually, I ended up spending a lot of the time organizing a release and writing process documents, so I didn't escape management as much as I would have liked.

Now I'm back, planning on doing the Twisted 10.0 release and working with didrocks to get better Launchpad integration with Quickly, as my contribution to Project Awesome Opportunity.

Twisted 10.0.0pre1 released

2010-02-22T13:51:00.002+11:00

It's not going to work, it's not the final release, but I'm very pleased to announce that Twisted 10.0.0pre1 is available for testing.

Download it, test it, play with it and help us make 10.0 the best release ever!

On a side note, this is hopefully the start of a much simpler, more automated release process and maybe even time-based releases.

From the Strategist

2010-02-11T01:00:00.002+11:00

If I don't dump a bunch of stuff that's on my mind, I'll never blog again. Here goes:

Launchpad users within Canonical seem to appreciate the Roadmap. Still receive some complaints about not communicating our plans enough, but not sure what to do.
Team lead meeting in London went well. Good to see the gang again.
The Wellington sprint was amazing. Rarely seen a sprint that's more productive.
My favourite result: a fixed, pre-planned annual schedule of sprints, including two annual whole-team sprints. I get to meet with my colleagues and have a life and Canonical saves money all at once.
The team leads like the Ready-to-Code and Launchpad Enhancement Proposal process things.
Daily builds work is going well. There's a lot of UI bits to consider, and we're trying to get the UI correct before building it. Grates my JFDI nerves, but maybe it's the right approach.
Challenged the Launchpad team leads to blog once every two weeks about anything, as long as it's vaguely related to Launchpad. The name of the challenge? Blog like it's 2006.
Community Help Rotation sucks and basically no-one really knows what to do about it.
I'm cautiously excited about using kanban to track our work-in-progress.
Launchpad needs graphs in the Launchpad application. I think all we need is someone to just add one. I don't have the time to do it as part of my day job, and I've got a backlog of other hacking tasks. Could I help you do it instead?
I'm going to PyCon.

Black out

2010-01-26T22:13:00.002+11:00

I've been a bit slack and have yet to black out my website or avatar to protest the Australian government's mandatory Internet censorship policy.

In case circumstances prevent me from doing so, please read the Internet Blackout Australia website to see why it's so important. You might also want to read Hugh Blemings' well-written thoughts on the matter.

Electronic book reader?

2010-01-25T18:16:00.001+11:00

On my other blog, I ask about electronic book readers. Recommendations greatly desired.

Python testing goodies

2010-01-05T11:07:00.003+11:00

Just found out that there's a PPA with the latest releases of a whole bunch of Python testing goodies, including:

testtools - Extensions to unittest that make real extensions possible
testresources - Safely re-use expensive resources in tests without paying massive set-up costs
subunit - Manipulate, exchange and analyze test results without writing code
testscenarios - Run the same tests against many different implementations

If you are running Ubuntu 9.10 or later, then sudo add-apt-repository ppa:subunit will add it to your apt sources.

We're starting to grow a lot of useful, small testing tools that become even more useful when combined. I really like the "small pieces, loosely joined" approach, but sometimes that can make deployment & dependency management a colossal pain. Happily, Ubuntu, Rob and Launchpad PPAs to the rescue.

testtools 0.9.2 released

2009-12-16T09:55:00.002+11:00

I've just released testtools 0.9.2, which I firmly believe is the best testtools release ever. Thanks very much to Robert Collins and Benjamin Peterson for making it so.

For the last few releases, we've been working on being more than just a simple aggregation of existing unit testing best practices and tried to do our own experimental extensions.

These extensions aren't actually all that exciting by themselves. We haven't added better logging support or new types of outcomes or test replay or smart rendering of error results or anything like that. I'm pretty sure testtools will never do those sorts of things.

Rather, the extensions we've added are designed to let you do that, and then share your extensions with other people without getting them into the standard library's base unittest classes.

If you are using testtools, you can change the way TestCase.run() works without overriding run and without figuring out how to safely call user code. You can handle exceptions raised from tests however you'd like — again not needing to change TestCase.run(). You can add new, rich types of assertions without having to modify some base class somewhere. You can store information on a test object that can be used by a sufficiently smart TestResult, which can be handy if you want to see, say, access logs for all failed tests.

Of course, all this starts to get really powerful when testtools is in the standard library, and all of the other major Python test frameworks inherit from it: nose, py.test, zope.testing and Twisted Trial.

Even so, it's worth switching to testtools today, just for the assertion logic alone. All it takes is changing the base class of your test cases to testtools.TestCase. If your test framework supports running standard Python unit tests, it'll support testtools.

Lean Lean

2009-11-28T03:50:00.003+11:00

Ever been confused by people saying "this is Lean" or "that is Lean" or "we should actually be using a cyclometric value-stream set-based burndown kanban for this"? I have, so I thought I'd write a quick pared-down cheat sheet to explain exactly what Lean is. A lean version of Lean, if you will.

At the very least, Lean is seven principles which, we're told, we should apply to making software.

Eliminate waste
Amplify learning
Decide as late as possible
Deliver as fast as possible
Empower the team
Build integrity in
See the whole

The first principle is fundamental.

List taken from Lean Software Development.

How to ask an open source project to do something

2009-11-27T20:12:00.003+11:00

This isn't exhaustive, but I think there are two ways to ask an open source project to do something:

I want X. You should do X.
I want X. How can I help do X?

I have no data, but I think that asking the second way is what makes open source actually work.

To do: To dos (Palm Prē)

2009-11-26T20:22:00.002+11:00

You might already know that I have a lot of lists on my computer. I may have mentioned once or twice that I have such lists. These lists are connected to a certain popular productivity system which isn't really about productivity and is a little bit like a cult.

I might have also mentioned that I've been longing for a "smart" phone, partly so I can have these lists with me at all times, to clutch at reflexively like some Popish talisman.

Now that I have a Palm Prē I have lists and a smart phone, and yet one does not know the other. I am at a loss as to how to get my lists — line-separated text files on my laptop — onto the Tasks application on the Palm. Even then, once they are on the Palm, I have no idea as to how to get them synchronized with something I can edit from my laptop.

Any ideas?

Palm Prē reflections

2009-11-24T23:31:00.002+11:00

I got a Palm Prē yesterday. Here are some initial thoughts.

I like the feel of the physical device
The keyboard is small
Camera is useless for taking photos of receipts
The gesture UI is great
The Mail application shows individual emails, rather than GMail-style conversations. Lame.
I don't know how to get my lists out of text files on my computer and onto the Palm
I don't know actually know anything about syncing from the Palm
I still don't regret not getting an iPhone

UDS

2009-11-23T22:48:00.002+11:00

I'm back in London after a mammoth two weeks travelling the globe bringing Launchpad joy to its four corners.

First up I went to Mooloolaba, Australia. If you're Australian, you probably know how to pronounce it. If you aren't, you'll probably never get it. Sorry.

I was there with the Canonical Bazaar team and some other Launchpadders, talking about their next few months of development. They are going to be focusing on helping Launchpad and the Ubuntu platform team work on our daily build initiative. Hmm. I really need a place to link to for that last one, don't I?

Mooloolaba is a beautiful place to be in November. If you are thinking of giving me something for Christmas, consider a holiday apartment there.

After Mooloolaba, I crossed the Pacific to reach Dallas, home of the twenty-eight lane highway. I didn't really get into the city that much, but man, UDS was great.

Most of the people there are directly involved in Ubuntu development. It's such a joy to see so many people from all over the world come together to work on Ubuntu, and an honour to think that I have a place in that work.

I spent most of the conference flitting from room to room, going to sessions about the Ubuntu development process, daily builds and the Software Centre. Occasionally, I cheated a bit and went to a random session that interested me. Mostly it was about Quickly.

As a Launchpad developer, one of the most exciting things about this UDS was seeing that Ubuntu developers have started building their own tools around Launchpad using our API. Many of these tools are things we'd love to integrate with the main web app, but many are things that are so Ubuntu policy specific that we're happy to leave them outside. After all, this is exactly

what the API is about.

I did a short plenary session where I hacked up an API script using launchpadlib and demonstrated it live. After the presentation, many people came up to me and started talking about their API-using programs and their experience making them. The amount of interest really surprised me.

I also ran a couple of workshop / discussions. The first was on improving Launchpad for API use, which was a bit disorganized, since I didn't realize I was supposed to be running it. The second was on getting started with Launchpad hacking.

For the "Getting started with Launchpad hacking" session, I passed around a USB stick, and asked the audience for an API within Launchpad to expose. mdz suggested "blueprints", so I went to expose the 'drafter' element of a specification. In front of a room full of people. Sheesh.

If you think pair programming is hard (it is, and if you think it isn't you are either doing it wrong or you have extremely helpful colleagues), programming in front of a room full of people is harder.

Still, we managed to actually expose the API and propose the branch for merging with only a little bit of evil.

The point of the session was to get people set up with Launchpad development environments, to show them how to even begin to expose a method over the API but even more than that, to give people a little push into just giving Launchpad development a try. By blundering my way into a valid patch, I think maybe we succeeded.

Now UDS is over and I'm back in London, jetlagged and exhausted to be sure, but genuinely happy about the plans we have for Launchpad development, and eager to begin to make them happen.

Evil Overlord talks

2009-11-14T00:26:00.003+11:00

My "Hack Like an Evil Overlord" talks are now linked from my stuff page.

Tests: Costs and Benefits

2009-11-11T18:57:00.002+11:00

I like unit testing a lot. I've recently been thinking about whether it really is everything that advocates like myself make it out to be. I'm still thinking, but I reckon that language has something to do with it, and that I wouldn't like TDD so much if I were writing in Haskell. Maybe.

Anyway, here are the benefits of unit testing, without justification or explanation:

Easier to maintain code
Avoiding regressions
Make API clearer, both by describing it and by encouraging contracts.
Confidence in changes to code

For new contributors, there are other benefits:

Confidence in changes to code
Confidence in the validity of your own patch
Mechanism for exploring internal system behaviour

Some of the costs are:

Maintaining tests
Caring about the performance of tests
Running the tests
Debugging tests

For new contributors, there are other costs:

Finding tests
Figuring out how to run tests
Figuring out how to write tests
Learn unit testing

Is this fair? Is this clear? And exactly where was that post by that guy about unit testing being overrated?

On a half-related and slightly exasperated note, many of the conversations I've been having recently have been about costs and benefits. What happened to doing something because it was the right thing?

Raise your standards

2009-10-28T03:46:00.002+11:00

I gatecrashed LugRadio Live last Saturday and went to an excellent talk by Matthew Paul Thomas about reporting usability problems. Matthew has reported well over a thousand bugs against Launchpad itself (many of which have been fixed) so he knows something about the subject.

Matthew gave six pointers on how to report usability problems, but for me the single most striking one was this:

Raise your standards

When you are using an application with the aim of improving its usability, you need to shed all of your tolerance and forget those many workarounds you've learned. Drum your fingers impatiently while you wait for something to load. Think of how this would look if you were showing it to a sceptical friend.

The point struck me because so many times as an application developer I've thought to myself "it's good enough" or "it will work for now". Really though, that's not good enough. I should work until the thing I've added is invisible, or at least fun to use. I need to raise my standards.

Having raised one's standards, the trick is then to avoid the traps of perfectionism. Do you have any thoughts on how to walk the road, avoiding the pitfalls, standards held high?

Bug squashing sprints

2009-10-20T22:50:00.002+11:00

Every time somebody asks me what I think we should do for a sprint, I try to suggest "fix as many bugs as possible".

Each of the Launchpad teams gets together in a single location for a week long "sprint" fairly regularly. Maybe each team has two sprints a year.

Often these sprints end up being long talking sessions where designs for features are thrashed out. That's great and all, and sprints are a rare opportunity to do that, but often the design work goes unused for months while the team deals with the work already on its collective plate.

Sometimes these sprints have the whole team working (i.e. coding) toward a single milestone, be it a release or a major feature. These sprints are fun, but they've been quite rare for Launchpad. (Bazaar had a great one in Brisbane earlier this year).

What I'd love to see is a sprint where people fixed as many bugs as possible. Perhaps as a product strategist, I ought to be advocating higher-level visions and responding to the market and so forth, rather than saying "fix lots of bugs". But I don't think so, at least, not yet.

Fixing as many bugs as we can in a fixed time frame will make many users happy, since behind each bug is a user in pain. It will make us happy as a team, since a vastly dropped bug count will make us feel less overwhelmed and will feel like a concrete achievement. It's an easy sprint to measure the success of, and my hunch is that it would be a fun sprint too. It's also substantially different to what we normally do, which adds to the fun.

What do you think? I'd love to hear from people in community-driven open source projects, as well as people within Canonical who aren't in Launchpad. Maybe we're missing something that others know about.

Command line apps

2009-10-15T21:42:00.003+11:00

Back in the day, I used to want to use twisted.python.usage for all of my command-line apps. It's a fairly nice API & a good way of writing code.

Since I started writing Bazaar plugins though, I've fallen in love with Bazaar's command interface. Jamu has too, so he wrote Commandant. I don't really understand it though, which is mostly for lack of trying.

Michael turned our internal EC2-using testing tool into a bzr-a-like app using the Bazaar APIs. There's a bit of duplication, but it works pretty well.

The Bazaar developers are aware that their command-line interface code rocks and that it's too closely bound to Bazaar. I'm not sure whether anyone has filed bugs about it or not, but here's what I actually want:

The subcommand interface, e.g. "bzr foo"
Option parsing
Help
Command aliasing
Error handling
Progress display

There's also a bunch of cool stuff in Bazaar that's useful for a lot of applications, command-line or not.

Registries
Transports
Hooks
Oh yeah, plugins. (How could I have forgotten this?)

Sometimes I wish projects like Bazaar and Twisted split this sort of stuff out into separate packages. That would probably change the way the packages are maintained, and I don't know whether it would be for better or for worse.

Sadly, no moral or action for this post. Just stuff that's been on my mind that I wanted to write down and publish. I'd very much welcome input.

Launchpad extensions

2009-10-15T20:16:00.002+11:00

Launchpad has a pretty awesome public API, implemented using lazr.restful. I've written a few small scripts for it, and the Launchpad team has a few scripts that they use internally for doing admin tasks.

The Ubuntu Platform team does a heap of stuff with the Launchpad API. James Westby has been using it to make sure that there's a branch on Launchpad for every single package in Ubuntu.

There's all this great work, but there's been nothing to tie the room together. I've seen hardly any discussion about how to write Launchpad API applications, or how to test them, or how to get launchpadlib working in GTK+. I haven't even seen much code sharing.

So, borrowing a trick from Twisted's tx super-project, I've created an 'lpx' project group on Launchpad. Bring it your scripts, your applications, your huddled masses. If you want to know more about the API, look at the API help page.

Thanks to Mark for reminding me that this is important.

Launchpad status now on identi.ca

2009-10-09T20:06:00.002+11:00

Sometimes, sadly, Launchpad doesn't behave as well as we'd like.

Although we try extremely hard to always be available, and take each instance of downtime (planned or unplanned) as a personal insult, sometimes we're forced to disappoint both our users and ourselves. It's no excuse, of course, but at least we're in good company.

To make these inevitable interruptions easier to swallow, we've set up identi.ca and Twitter accounts that report the Launchpad server statuses, and only the server statuses. We've got people all around the world who are able to update it when things go bump in the night. There's also a link to the identi.ca page in our footer, so you don't have to keep consulting a blog.

Thanks very much to Matt Revell for getting this done.

Meta-meeting stuff

2009-10-09T05:19:00.002+11:00

Last week the Launchpad team leads gathered in London and had an absolutely huge meeting. Fifteen or sixteen smart, opinionated & passionate people in a room talking about the next six months of Launchpad development. It was a lot of fun.

When planning the agenda, Martin Albisetti and I were a little worried about things wandering off track, so we thrashed out a lists of dos & don'ts. I'll blurt them out here, with the really terrible ideas filtered out so I look more clever, and then offer some thoughts on how they worked.

Do

be happy with the results of this meeting.
ask questions, especially "why"
be punctual
stay on point

Don't

design features
check your email, i.e. laptops closed
leave the meeting without being clear
leave the meeting without being happy

The "laptops closed" one didn't work so well. It was great for the first couple of days, but by Thursday afternoon there were always people in a corner with those open LCD screens sucking chi out of the room like little dragons of despair.

The "Don't design features" rule was awesome. The rationale is that feature design leads to long discussions, and that it does a disservice to the many excellent engineers we have who aren't at the meeting and will be actually implementing a feature. At best, it's waste and at worst it's us telling people how to do something that they know how to do better. The clearest sign that it was a good idea came when other people started using it to shut me up.

Talking Time

2009-10-08T02:44:00.004+11:00

I've been working on the Launchpad team for a while, with most of that time being in Australia. Others in the team are in the US (red states & blue states), the UK, Germany, Brazil, Canada, New Zealand, Serbia, Lithuania and Thailand.

Here are some tips I've picked up for smoother online conversations, particularly around scheduling.

Don't say "summer" or "fall". Say the month, or the range of months.
Say "my morning" rather than "the morning".
Use 24-hour time.
In real-time conversations, say "in 2 hours time" rather than "at 10".
Always include the timezone. Avoid using local abbreviations (e.g. PST), instead use offset from UTC.
Better still, just give the time in UTC.
Know your UTC offset.
Use a timezone-aware meeting planner. You'll get the arithmetic wrong otherwise.
Say "Oct 7" rather than 10/7 or 7/10. Everyone speaks English, but not everyone uses your dialect.
The time that you end a meeting is more important than when you start it. Thoughtfully consider the timezones of other attendees when you are planning for & participating in meetings.

Violating these rules isn't a big deal, since people can generally figure out what you mean. Following them, however, can speed things along and sometimes even avoid tedious conversations.