Tired Hippo

Chronomancy... is generally a fictional and sensational school of magic. Although the school is based on quantum physics and certain scientific theories, there is no concrete evidence of the perfected use of time manipulation.

Thank you Wikipedia.

Let the Beat Drop

Didn't get around to posting last weekend, sorry. Next weekend I'll be at a conference in the north of Tasmania, so no post then either.

Things in the works:

• Automated testing as a community builder


• Asynchronous functions


• Asynchronous development methodologies

Also, glyph thinks that xUnit sucks, but hasn't bothered to explain why. So I want you to:

1. Tell me why xUnit sucks


2. Hassle glyph until he does 1.

Registration disabled

You don't need to register to comment any more.

7

My headphones broke and I'm chasing down an erratic test failure with no real leads.

And yet Prince stands:

All 7 and well watch them fall
They stand in the way of love
And we will smoke them all
With an intellect and a savoir-faire
No one in the whole universe
Will ever compare
I am yours now and u are mine
And together well love through
All space and time, so dont cry
One day all 7 will die

Living for the City

Isn't talking shop fun? I know there's more to the world than computers and programming and blah blah blah. Sometimes photography, the election, real estate and the rediscovery of seventies pop music need to be set aside to make way for a serious, meaty discussion on anonymous creation methods.

I've been up in Sydney for a couple of weeks, working with the other Canonical guys. I hadn't realised how much I missed working with other hackers in the same room.

Sydney has good things besides programmers, of course.

Not exactly sure when I'll be moving, but it'll happen eventually.

Within You Without You

Testing is hard, writing testing frameworks is easy. In an effort to make testing easier, big projects like Twisted, Bazaar and Zope write their own testing frameworks. That way they control both the test runner and the tests that are run. It's actually quite convenient.

However, it's led to a significant problem:

There are many similar implementations of xUnit in Python, each with subtle incompatibilities.

Running Twisted tests in the Zope test runner? Watch out for the threads that the reactor maintains between tests. Running Bazaar tests with Trial? On my machine, I get told that elementtree doesn't have an 'ElementTree' attribute. Hmm.

When talking about this problem, I often refer loosely to "PyUnit compatibilty". The idea is that:

1. Every Python test runner should support running vanilla Python standard library unittest.TestCase tests.


2. Every Python unit test should be able to be run using the mechanisms in unittest.py in the Python standard library.

In other words, this code should Just Work:

import unittest
from yourframework import testing

class PythonTestCase(unittest.TestCase):
    def test_something(self):
        pass

class FrameworkTestCase(testing.TestCase):
    def test_something(self):
        pass

if __name__ == '__main__':
    python_test_result = unittest.TestResult()
    framework_test_result = testing.TestResult()
    FrameworkTestCase('test_something').run(python_test_result)
    FrameworkTestCase('test_something').run(framework_test_result)
    PythonTestCase('test_something').run(python_test_result)
    PythonTestCase('test_something').run(framework_test_result)
    # At this point, python_test_result and framework_test_result
    # should hold equivalent data.

If your framework is PyUnit compatible then the above fragment should give the same results if run directly or if run in your runner. Things get a little bit hazier when it comes to test discovery.

So, if your unit test requires that it be run inside a special suite (e.g. TrialSuite) in order to work correctly, it is not PyUnit compatible. If your test runner does some critical set up that enables features that your tests need, then it is not PyUnit compatible.

This leads to a kind of thinking where certain features belong on the base test case and others belong in the test runner. Putting features in the wrong place might not lead to a strict incompatibility, but it can lead to significant inconvenience. (And what are automated tests if not a convenience?).

Two examples from Twisted:

Temporary Working Directory

Trial creates a _trial_temp working directory and changes into that directory to run tests. In Trial, this feature is provided by the test runner. It should be provided by the base TestCase class.

• It's not clear that every test needs this feature.


• Twisted tests now assume that they can create files with impunity. When Twisted tests are run in a different test runner, they leave garbage files everywhere.

Timeouts

By default, any Trial test that runs for more than two minutes will fail with a timeout error. The timeout period can be configured on a per-test, per-test-class, per-module or per-package basis. Trial implements this feature on TestCase, it should be implemented on the runner.

• Even in the Twisted test runner, this makes debugging more painful. You must do all of your debugging in under two minutes.


• Intuitively, you might think that the runner should control how tests are run.


• Tests that don't descend from Trial's TestCase can still hang.


• Two minutes might be good enough for me on a Monday, but I might be busier on Friday. I should be able to change the timeout without changing code.

That's Not Me

This blog started life as "Petty Pace". Blech. Too wanky and too quiet. "awesome radtastic ninjacore" is a stop-gap measure.

What do you reckon should be the name for this blog?

Heartbeats and Sails

Mark Shuttleworth:

What’s good enough performance? Well, I like to think in terms of “heartbeat time”. If the major operations which I have to do regularly (several times in an hour) take less than a heartbeat, then I don’t ever feel like I’m waiting. Things which happen 3-5 times in a day can take a bit longer, up to a minute, and those fit with regular workbreaks that I would take anyhow to clear my head for the next phase of work, or rest my aching fingers.

Take this rule of thumb and apply it to unit tests:

• Tests for whatever chunk of code you are working on should take "less than a heartbeat".


• Your entire testing suite (that you run 3-5 times in a day, right?) can take longer to run, up to a minute.

Authors of tests and testing frameworks, there's your challenge.

Tests that take too long to run just won't get run. Programmers will postpone running the suite until the last possible moment. When using something like PQM or Buildbot, this can be disastrous. Other developers might have to wait hours for their code to land on trunk.

Gerard Mezsaros's new book, xUnit Test Patterns has some good ideas about what to do and what not to do to make your tests run in a couple of heartbeats.