- It's dull. Really dull.
- You find bugs, but it's somehow more frustrating. Perhaps because you thought your code was correct already.
- The code is probably not written for testability, which means you have to mix refactoring up with verifying behaviour. Messy & perilous.
- Alternatively, you write tests with a lot of mocks. Not bad in itself, but risky.
- It's much harder to get full coverage.
- You write tests for things that you don't care about, just to exercise a particular code path. This makes the tests more fragile.
- You never really know when you are finished.
Are you a TDDer or a code-and-cover person? Why do you prefer it that way?
9 comments:
I've been flip-flopping a lot on this. I probably do more TDD than not, but there are still scenarios where I just can't start off with tests. Usually, it's because I either don't understand the problem well enough yet, or because the testing infrastructure work needed is pretty big, and I can't bring myself to spending so much time upfront on it.
The point that I enjoy the most about writing tests first is that it breaks the task down into a lot of mini-successes: write test, run code to show failure, add functionality, run test, success! For me it makes it a bit more like a game where you're always trying to reach the next level which is just out of reach.
Without writing tests first, it's more like a long wait until a (potentially) big success, followed by many small failure moments as you start finding bugs.
But I agree with Martin... when I'm unsure about which way to go, sometimes a tracer bullet helps me see how I can test.
I tend to be a "code & cover" type, mostly because the environments and systems I write for are hard to simulate in a test and I often am not sure what I need to do beforehand or how to do it -- i.e. prototyping work.
Not very good excuses, I know.
The big exception to the rule for me is a prototype, which then turns into something useful. There's no way I am going to write tests up front for a prototype.
This seems like it could use a little expanding; it seems to be written for an audience that mostly understands what you're talking about already, and will just evoke familiar pangs rather than really educate. Why is code-and-cover dull? You write the same tests and code either way, right? Why is it harder to get full coverage? You just write your tests and then run your coverage tool, of course.
I think that there are a lot of folks out there who do code-and-cover but don't really know why it's not as good as TDD. A more expansive post would benefit them quite a bit.
(Of course, I know the answers, as do many of your readers, but I'm too lazy to write a good post in response, so I'm hoping you'll do it for me ;-)).
For my part, I do sometimes have trouble getting tests first, mostly in not-fully-covered code that has a big pile of existing dependencies that don't have test mocks, and building out even the most trivial test infrastructure would take substantially longer than just fixing a simple bug. However, when I do get around to fixing it "for real", I try to avoid the code-and-cover mistake of writing a test that passes first: I cut the whole implementation into another file, write a test, and get it to fail first, then start re-importing the smallest bits I need to get my test passing. Once I'm in the TDD groove again, I will quite often spot uncovered or buggy chunks of functionality and get them fixed right.
The few times I have tried to just write tests for existing code without modifying it so that it will fail, it's been a complete mess.
... and another thing.
There is one case where not writing tests first is actually a better idea, I think.
If you have some code with a lot of dependencies and no test support for those dependencies (where "test support" means mocks, fakes, stubs, in-memory implementations, or what-have-you), sometimes TDD means that you have to start off by writing a largeish pile of unmaintainable one-off junk just to stub out enough to get a simple test running.
I go back and forth on this, but I am coming to be of the opinion that having big piles of unmaintainable test stubs can be a worse problem than having poor coverage. The stubs and functionality and interfaces can diverge from the "real" implementations, and inevitably you end up having to maintain a couple dozen fake implementations of the not-really-test-supported interfaces, each of which has its own quirks. This can lead to lots of false failures, which leads to decreased trust in the test suite, which is of course bad.
If you're adding functionality to a system, you can usually manage this problem by implementing the new functionality itself in a corner, with only the dependencies that it really needs, and keep the test maintenance burden sane. Rather than try to make sure that the integration code is properly TDD, the goal is then to just keep the integration code (which glues your shiny new TDD-developed module into the system) as small as possible.
This is really just a restatement of "minimize untestable code", but I take issue with that phrasing, because it's defeatist (especially in a nice, dynamic language like Python). All code is testable: this strategy should be a temporary measure as you work towards developing good test support for your "untestable" interfaces and thinning out unnecessary coupling. I think that it may be worthwhile to do some planning and architecture around your test development as well as your main body of code.
Whether you do TDD or code-and-cover it's very important to make sure your test fails when the code you think it is testing is broken (or gone). Doing TDD ensures this fail-when-broken check happens for each test. This validation can be performed with code-and-cover, but it is much more tedious.
I prefer TDD but use code-and-cover when I really don't understand what I need. In that case, the code-and-cover I do is to (1) spike until I know what I want and then (2) comment out my code and TDD until it's all uncommented and covered properly.
Thanks for the comments everyone.
Julian, if by prototype you mean "proof of concept", then I also often don't do that TDD. What tends to happen with those is that the prototype becomes the tests. (After all, how did I know it worked?).
Glyph, I would love to write up a more detailed version of this post. It's unlikely I'll have the time.
A thing that can help with untested mocks & stubs is "interface verification tests": one set of tests that runs against the real thing and against a more easily testable thing.
I tend to sketch out a minimal "spike solution", then write some tests for it, and then use TDD to add functionality and make sure corner cases get handled.
Post a Comment