Robert Collins of Bazaar, recently blogged about the cost of slow tests. I couldn't agree more.
Launchpad has a very slow test suite. It takes upwards of five hours to get a change into our "known good" branch. It's been a source of pain for the team for years, and now that Launchpad is Free Software, it's becoming a pain to new contributors.
We've tried a few things to deal with it, none of them great. First up, we've got a tool for running the test suite on EC2 instances. It's nice, but it's still too slow. I've written a "faster tests" spec outlining some of the options we have; but specs never got software written. Likewise, there are a bunch of bugs flagged as build-infrastructure, many of which address the slow test sped. We also have a rotating "Build Engineer" position within the team to address those bugs.
It's probably not enough though. From watching Rob's blog posts, and the emails to the Bazaar mailing list, I think that what Launchpad really needs is someone with a passion for solving the problem (i.e. making Launchpad hacking fun) who is willing to lead the way, and a full commitment from the rest of the team for getting the runtime down and keeping it there.
More on this later.
Twitter Updates
Blog Archive
-
►
2010
(68)
-
►
November
(14)
- testtools manuals
- Tests that print stuff
- Boiling kettles, unit tests and data
- Big or small?
- "Don't Make Me Think", thoughts for Launchpad
- Reviewing specs, rock on!
- Having an extra feature never hurts, rebutted
- And then what?
- Make it really easy to fix bugs on Ubuntu
- What else have you got?
- Ubuntu in a VM on OS X?
- Still going
- What to do, what to do
- Launchpad and UDS-N
-
►
November
(14)
2 comments:
5 hours! Wow!
5 minutes is too much. To run some tests between code edits, 50s is really pushing it.
My preferred method of development is to run the one test or TestCase that exercises the code I'm working on, and trial has no problems doing that. When I think I'm fine with that, then run the entire test suite. Does Launchpad lack a good way to do that?
If I can't make-a-change-run-a-test in about a 10 second cycle, there's something wrong with my development environment and I won't rest until it's fixed ...
We're not completely stupid, of course we have a way to limit the tests we run between code edits :) Although this tends to also be too slow because of excessively general fixture setup...
Post a Comment