Launchpad improving GNOME's Bugzilla

Just read a very cool article about Canonical funding improvements to GNOME Bugzilla, making it easier for Ubuntu and GNOME to share bugs.

New sexy

My new shiny iMac 27" arrived today. It is very beautiful, although the screen is dauntingly large. When the Kanex XD thingies come out, I'll be able to plug my XBox into it too.

It's the first time in a long time I've done a fresh install of a non-Ubuntu computer though. Some things, however, remain the same.

1. Download decent browser (Firefox or Chrome)
2. Download IRC client (for OS X, Colloquy)
3. Generate SSH key
4. Upload SSH key to server, register it on Launchpad
5. Change the default Terminal settings to light grey text on a translucent black background
6. Get Emacs

Oh, and to forestall questions, I'm not going to dual boot right now but I plan to eventually.

Cowtasks

If you've got a Palm Pre and you want to be able to synchronize tasks from the Pre to Remember the Milk, then please start hacking on CowTasks.

The original author has given up development, open sourced it and I've put it on Launchpad. I don't have the time to maintain it either, but I'd love it if someone did.

Visualizing yak-shaving

Over the weekend I read Cary Millsap's excellent paper Thinking Clearly About Performance. In it, he hijacks the UML Sequence Diagram format by drawing the vertical axis to scale. The bigger the gap between lines, the more time was taken performing that interaction. Suddenly, the sequence diagram can give you a clear picture of where time is being spent.

Wouldn't this be a useful diagram to draw instead of value-stream maps? You would lose the easy distinction between "value generating" and "waste" steps, but I think you'd gain a deeper understanding of hand-offs and of system complexity. Drawing it to scale would give a helpful picture of where time is being spent, and where response time can be improved.

More generally, Lean folk should probably read Millsap's paper and think hard about what it means for software engineering process.

Also, I reckon drawing up a quick sequence diagram would be a great way of seeing how deep one is in yak-shaving. Each time you go one layer deeper, add a new column and draw a new arrow and make a note of when & why.

launchpadlib gotchas

I've shown you how to get started with launchpadlib and have shown a slightly more complex launchpadlib script.

Before I cry The power is yours! and return to my home within the earth, I'm going to warn you about the things that can trip you up when using launchpadlib.

Bugs

launchpadlib has bugs. There are also bugs in lazr.restful and lazr.restfulclient – two libraries that are core to launchpadlib's behaviour.

Welcome to software engineering.

Documentation

It's on the wiki, and it's good, but it could always be better. There's a page of as well as a guide on launchpadlib.

The reference documentation isn't written for Python programmers. It's written for REST programmers. Actually, it's not written at all but rather auto-generated from our source code. Sometimes this can be confusing, and I frequently find myself consulting the Launchpad source code to get things done with the API.

Error messages

I'm told this has got better with recent releases, but often when you get an error in launchpadlib, it looks like an HTTP error and you have very little help on how to debug it. Unfortunately, I don't have an example ready.

If you come across an error like this, file a bug and head straight to #launchpad-dev on freenode to get help.

Potato programming

It's really easy to write code with launchpadlib that does this:

  for thing in bunch_of_things:
    thing.do_something_on_launchpad()

Code like this is really slow. It will do one round-trip per thing, which can be quite expensive. Twisted folks sometimes call this potato programming.

Exposure

Not all of the code within Launchpad is exposed through the API. We have to expose things manually and we haven't done it all yet. Sorry.

If you come across something that you want, then please file a bug and tag it with api.

In general, exposing something of the API is really easy or almost impossible. If the thing you want falls into the first category, you can probably patch Launchpad yourself.

Testing

Testing launchpadlib apps is hard. You do not want your unit tests to run against launchpad.net and running your own instance of Launchpad simply to run unit tests is masochistically stupid.

I think the situation here has improved recently too, but I haven't heard much about it or explored it myself.

Conclusion

There you have it, all of the gotchas for writing code with launchpadlib. As you can see, it's not really any worse than writing for any Python library – I'm just being up-front with you because I like you.

If any of these gotchas no longer apply, please correct me and I will shout your good news from the rooftops.

Until then, happy hacking.

Have you tried lptools?

Have you tried lptools?

It's not at all officially associated with the Launchpad project, of course, but it's got a few nice things that you might want to look at, including:

• a milestone manipulator
• a code review notifier, and
• a milestone to iCal exporter

One of these days, I'd like for there to be an official, awesome Launchpad command-line client. As it is, I'm happy with the world of people making extensions to meet their needs.

For other extensions, check out the lpx project or our list of clients.

launchpadlib powerup

In my last post, I introduced launchpadlib and demonstrated a very simple script that uses it. In this post, I'd like to build on that a bit and show you how to do something actually interesting.

In particular, I want to show you how to search for bugs, teach you a bit about Launchpad's internal data model and help you help yourself when it comes to figuring out Launchpad APIs.

The script at lp:~jml/+junk/bugstats is designed to tell you how good you are at filing bugs. It uses a very simple metric: out of the bugs that you've filed, how many actually have been fixed.

$ ./bugstats.py ubuntu jml
jml is 22.22% successful on bugs in Ubuntu
$ ./bugstats.py launchpad-code jml
jml is 47.63% successful on bugs in Launchpad Code

To do that, we need to:

1. get the "project" and person referred to on the command line
2. search for all fixed bugs filed by that person
3. search for all bugs in total by that same person
4. count them both
5. divide them
6. print them!

I say "project", but I really should say "pillar", which is the Launchpad technical term for a project (e.g. "bzr"), distribution (e.g. "ubuntu") or project group (e.g. "gnome"). A pillar is anything in first part of Launchpad URL that isn't a person.

We get the pillar and person like this:

   pillar = launchpad.projects[pillar_name]
  reporter = launchpad.people[reporter_name]

Pretty easy, huh? Now, how do we search for bug tasks?

The first port of call is to go to the Launchpad API reference page. I'm going to look for the string 'reporter', since that's the one thing I definitely know I want to find.

Eventually, I found the searchTasks method (named operation) that's on pillars and takes a bug_reporter parameter and a status parameter. It returns a collection of bug_tasks, which are the objects that represent the rows in the table you see at the top of a bug page.

I can find the bugtasks for the bugs I've reported that have been fixed by doing:

   fixed_bugtasks = pillar.searchTasks(
      bug_reporter=reporter, status=['Fix Released'])

It took me a while to figure out exactly how to spell "Fix Released". I ended up using trial and error.

Similarly, I can all the bugtasks for bugs I've filed by doing:

   total_bugtasks = pillar.searchTasks(
      bug_reporter=reporter,
      status=[
          "New",
          "Incomplete",
          "Invalid",
          "Won't Fix",
          "Confirmed",
          "Triaged",
          "In Progress",
          "Fix Committed",
          'Fix Released'])

I cheated a bit for that one and looked at the launchpad code to get a list of all bug statusus. The default for searchTasks is to only return open bugs.

Once we've got the collections of bug tasks, we need to get their counts. In an ideal world, it would be len(total_bugtasks), but sadly bug 274074 means that len is really, really slow here.

Instead, I wrote this helper function:

def length(collection):
  # XXX: Workaround bug 274074. Thanks wgrant.
  return int(collection._wadl_resource.representation['total_size'])

With that, I can calculate & print my success rate at filing bugs:

   percentage = 100.0 * length(fixed_bugtasks) / length(total_bugtasks)
  print "%s is %.2f%% successful on bugs in %s" % (
      reporter.display_name, percentage, pillar.display_name)

Next up on the API, I'll talk about some of the gotchas and what you can do about them.

Get started with launchpadlib

In my spare time, I sometimes talk to people about how they can get started with launchpadlib hacking.

launchpadlib is the Python client-side library that talks to Launchpad's own REST API. It turns out that customize scripted control of a bug-tracker-code-hosting-translation-distribution-building-cross-project-collaboration thing is actually quite handy.

If you want to get started hacking with launchpadlib, and you have Ubuntu, then install 'python-launchpadlib' now. I'm pretty sure you can also get it from PyPI.

You can check that it works by running:

$ python
>>> import launchpadlib
>>> launchpadlib.__version__
'1.5.1'

I'll be assuming you're running 1.5.1 or later.

I've written a very simple launchpadlib application that you can get with 'bzr branch lp:~jml/+junk/bugstats'. Each revision shows a meaningful launchpadlib script. You can get at the old revisions with 'bzr revert -r1' or 'bzr revert -r2' or '-r3'.

Here's what the simplest launchpadlib script that I could think of looks like:

import os
import sys
from launchpadlib.launchpad import Launchpad, STAGING_SERVICE_ROOT

APP_NAME = 'jml-bug-stats'
CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache')
SERVICE_ROOT = STAGING_SERVICE_ROOT

launchpad = Launchpad.login_with(APP_NAME, SERVICE_ROOT, CACHE_DIR)
print launchpad.bugs[1].title

(Adapted from r2 of the above branch).

A few points.

• We use STAGING_SERVICE_ROOT, which means that we're pointing
  at Launchpad's staging service,
  just in case we screw up any data.


• We give the application a name, when you run the application, launchpadlib
  opens up a browser window letting you decide how far the application
  can act on your behalf.
• We provide a cache directory. Credentials, among other things, get stored
  here.
  
• We then login and get an object that represents a Launchpad instance
• Once we've got it, we look at the collection of bugs, get Bug #1 and then
  print the title

Very simple. To learn how to write this application, I looked at the main Launchpad API help page, the
examples page and the reference documentation. You'll notice that I had to translate the reference documentation from REST-speak into Python-speak.

Already you have enough to go exploring with the Launchpad API and think of cool things to do. A bunch of people are already doing cool stuff and there are many projects that use launchpadlib.

Next up, I hope to show you some more complex things you can do with the API.

Monitor & keyboard

I want to get a shiny new monitor and keyboard. The monitor is for coding, writing docs, answering emails and watching films. The keyboard is for same. They'll both plug into my Thinkpad X200 for now.

I'm tempted to get an Apple Cinema Display (but at such cost!). I don't know what keyboard I want. I think that the DAS keyboard I have isn't it.

Thoughts?

Back from PyCon

I had a great time at PyCon 2010. The best part was definitely the Twisted sprint, which gave me a long overdue opportunity to do some actual coding.

Well, actually, I ended up spending a lot of the time organizing a release and writing process documents, so I didn't escape management as much as I would have liked.

Now I'm back, planning on doing the Twisted 10.0 release and working with didrocks to get better Launchpad integration with Quickly, as my contribution to Project Awesome Opportunity.