I just attended Test-Driven Development presentation which represents everything that is wrong about the way Agile advocates are trying to evangelize their practices. I don’t have anything against the presenter in particular, but it’s really time for Agilists to rethink the way they communicate with the real world.
Here are a few comments on his presentation.
One of the first slides that deeply troubled me claimed the following:
- Tests are (executable) specs.
- If it’s not testable, it’s useless.
First of all, tests are not specs. Not even close. Somebody in the audience was quick to give a counter-example to this absurd claim by using a numeric example ("how do you specify an exponentiation function with a test?") but my objection to this claim is much broader than that. Relying on tests as a design specification is lazy and unprofessional because you are only testing a very small portion of the solution space of your application (and of course, your tests can have bugs). Tests also fall extremely short of having the expressiveness needed to articulate the subtle shades that a real specification need to cover to be effective.
This claim is part of a broader and more disturbing general Agilist attitude that is usually articulated like "Your code is your spec", along with some of its ridiculous corollaries such as "Documentation gets out of date, code never does".
Anyone who claims this has never worked on a real-world project. And I’m setting the bar fairly low for such a project: more than five developers and more than 50,000 lines of code. Try to bring on board new developers on this project and see how fast they come up to speed if all they have to understand the code base is… well, just code. And tests.
I am currently getting acquainted with a brand new project that is not even very big, and while I understand Java fairly well, there is no doubt in my mind that for ten minutes I spend trying to understand how a certain part of the application works, a five-line comment would have given me this knowledge in ten seconds.
The second claim, "If it’s not testable, it’s useless" is equally ludicrous and a guarantee that at this point, the audience you are talking to is already looking at you as a crackpot.
Software is shipped with untested parts every day, and just because it’s not entirely tested doesn’t mean it’s bad software or that the untested parts are "useless".
Agilists just don’t understand the meaning of calculated risk.
Early in the development cycle, it’s perfectly acceptable to go for a policy of "zero bugs" and "100% tests". But as the deadline looms, these choices need to be reconsidered all the time and evaluated while keeping a close eye of the final goal. Very often, Agilists simply forget that their job is to produce software that satisfies customers, not software that meets some golden software engineering scale.
Anyway, let’s go back to the presentation, which then proceeded with the implementation of a Stack class with TDD. Before spending thirty minutes on a live demo of the implementation of a Stack class (are you impressed yet?), the presenter warned the increasingly impatient audience that they should "not pay too much attention to the Stack example itself but to the technique".
And that’s exactly the wrong thing to do.
Look, we "get" TDD. We understand it. Frankly, it takes all of ten minutes to explain Test-Driven Development to a developer who’s never heard of it: "Write a test that fails and doesn’t compile. Make it compile. Then make it pass. Repeat".
The hard part is applying it to the real world, and showing the implementation of a Stack will soon have everyone leave the room with the thought "Cute, but useless. Now let’s go back to work".
It was even worse than that, actually: The presenter kept taking suggestions from the crowd but he declined all those that didn’t fit in the neat script that he had in hands at all times. These suggestions were good, by the way:
"What should we test now?"
"How about: if we pop an empty stack, we get an exception"
<a bit embarrassed> "Mmh, no, let’s not do that" <hand waving, look down at notes and proceeds while happily ignoring the other raised hands>
To be honest, I am becoming quite suspicious of Agile practices for that reason: all the presentations I have attended and books that I have read are always using toy implementations as examples. Stack, List, Money, Bowling… enough already! Let’s talk about TDD for code that interacts with clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place (and: yes, I read Michael Feathers’ book, it has some good and some bad, but it’s not germane to Java and TDD so I won’t expand on it here).
And please, avoid smug and useless answers such as:
"A lot of the classes I have to test are hard to isolate, do you have any advice regarding mocks?"
"Well, if you had started with TDD in the first place, you wouldn’t be having this problem today".
Fundamentally, I am disturbed by the Agilists’ dishonesty when it comes to presenting their arguments. They offer you all these nice ideas such as Test-Driven Development and Pair Programming but they never — ever — disclose the risks and the downsides. To them, Agility is a silver bullet that is applicable in all cases with no compromises.
The truth is that these practices come at a price, and for a lot of organizations, the price gets high very quickly. Agile development will never go far if its proponents keep ignoring these organizations and make condescending comments to its members.
I like Test-Driven Development. I really do, and I’m fortunate enough to work on a project that lets me use TDD most of the time. But the truth is: at times, I don’t do TDD because implementing a feature quickly is more important than a fuzzy feeling. And I’m also aware that TestNG is an open source project with less than five developers, all of them on the bleeding edge and aware of the latest advances in software engineering.
And this is my main beef with Agilists: I strongly suspect that most of them are spending their time on open source projects with like-minded fellows, but none of them have any experience what companies whose survival depends on shipping software have to go through to organize huge code bases growing thousands of lines of code every day under the combined push of hundreds of a developers, all with their personal background, education and bias.
So here is my advice to Agilists: get real now, or you will become irrelevant soon.
Update: four years after I posted this article, reddit picked it up.
#1 by Sam on June 7, 2006 - 11:38 am
Have you ever worked on a *real* Agile project? I don’t mean the fake shit that most people refer to as “Agile” or “XP”, etc. I mean the real deal where people actually follow the prescribed principles without comprimising it to hell. I highly doubt it since those projects are, in truth, very rare.
Here’s some truth for you Cedric. I worked on a *real* XP project. We did the full deal — no bullshitting around like most “XP” projects. I started on the project very skeptical of the whole thing since I’d never done XP before. At the end of it, I was completely convinced of its efficacy. It was the most productive project I’ve ever worked on. Most of the projects I’ve been on don’t even come remotely close to that one. And yes, it was for a REAL project — not some made up bullshit to make XP look better.
My point is that until you do it, you have no business talking shit about it. There’s a reason why Agilists sometimes have an elitist attitude: it’s because a properly run Agile project will kick ass over the traditional processes with ease and they know it because the’ve ACTUALLY DONE IT.
#2 by mmatt on June 7, 2006 - 12:05 pm
Sam, that’s all shit, too. (And extremely close to the “Well, if you had started with TDD in the first place, you wouldn’t be having this problem today” comment from the presenter.) It’s that same dogmatic BS that Cedric is slamming–and rightly so.
Putting yourself on a pedastal elevates you out of the real world trenches. I’ve had the same experience with super-awesome XP projects, but have also seen the same practices fall apart on the next project with the same exact team in place with the same exact practices. When I read your comments (especially with the added emphasis on ‘real’), I think noobie, ignorant, or boutique developer. The minute you try to push your crap around in a larger, slow-moving company, you’ll hit so many roadblocks, you’d likely quit.
#3 by pixel on June 7, 2006 - 12:06 pm
Sam, that’s all shit, too. (And extremely close to the “Well, if you had started with TDD in the first place, you wouldn’t be having this problem today” comment from the presenter.) It’s that same dogmatic BS that Cedric is slamming–and rightly so.
Putting yourself on a pedastal elevates you out of the real world trenches. I’ve had the same experience with super-awesome XP projects, but have also seen the same practices fall apart on the next project with the same exact team in place with the same exact practices. When I read your comments (especially with the added emphasis on ‘real’), I think noobie, ignorant, or boutique developer. The minute you try to push your crap around in a larger, slow-moving company, you’ll hit so many roadblocks, you’d likely quit.
#4 by Tim on June 7, 2006 - 12:35 pm
There is a distinct lack of agility displayed by many Agilists these days. This is just another example of it. See http://pliantalliance.org for my backlash.
#5 by Agile Thugz on June 7, 2006 - 12:37 pm
Hizzy you ever worked on a *real* Agile project? I dizzon’t mean tha fakes S-H-to-tha-izzit tizzle most thugz refa ta as “Agile” or “XP”, etc. I mizzle tha rizzle deal where thugz actually follow tha prescribed principles witout clockin’ it ta hizzle fo’ sheezy. I highly doubt it since those projects are, in truth, very rare.
Here’s some truth fo` you Cedric from tha streets of tha L-B-C. I worked on a *real* XP project. We did tha full deal — no bullshitt’n around like mizzy “izzy projects . Chill as I take you on a trip. I started on tha project vizzle skeptical of tha whole thing since I’d neva done XP before . Im crazy, you can’t phase me. At tha end of it, I was completely convinced of its efficacy. It was tha mizzy productive project I’ve ever worked on with the gangsta shit that keeps ya hangin. Most of tha projects I’ve been on don’t even come remotely close ta T-H-to-tha-izzat one mah nizzle. And yes, it was fo` a REAL project — not some made up bullshit ta makes XP look betta
#6 by Alex on June 7, 2006 - 1:01 pm
IMHO, your post would be best served by replacing the term “agilists” with “some TDD book authors and speakers”. Having attended many of the agile conferences and countless user group meetings, I hear you. Not that you won’t find the exact same thing at OOP, BDUF, SQA, etc conferences…
I have a lot of respect for what Uncle Bob, Michael Feathers, Brian Marick and others have accomplished and pursued. As in increasing the lines of test code written by programmers that used to look down their noses at it. I’ve been at several companies where this has improved thanks to these folks and others. Not that our little silicon world is perfect, but it is a better place now with bad “test engineers” replaced by better “extreme programmers”.
Lets not forget that *a lot* of the folks your post refers to as “agilists” spend most/all of their time writing better code and tests — and not going around and talking about it. Coding solutions for real problems is more fun than presenting Stack examples!
Keep up the writing and sharing your perspective. Agile’s failures and mistakes won’t kill us, they’ll only make us stronger
I still hope you find yourself in Colorado to speak for our “agilists” group — maybe a panel “Otaku vs Langr”
#7 by fanguad on June 7, 2006 - 1:29 pm
I’ve never worked on a “real” Agile project, but I do know one thing – Agile isn’t a panacea, not even close. I work on a project that *can’t* be Agile. We have hundreds of developers spanning dozens of different companies and divisions. Many of the things I’ve seen Agilists attribute to themselves (lots of tests, good customer interaction) are more the hallmarks of a good team, rather than methodology.
When someone else on this project inevitably needs to understand my code, “self-documenting code” and unit tests aren’t going to cut it. Likewise, they’d be equally unhappy with obtuse code lacking any form of testing. Any shortcuts are going to make that person’s job harder. Agile promotes certain shortcuts, while (for example) waterfall promotes others.
I’m going to make an appeal to the moderates here: pick the tool that does the job. For some projects, Agile is appropriate, for some it isn’t. For some (and here I’m covering my head) it probably doesn’t matter what technique you pick if you have good people.
#8 by matthew on June 7, 2006 - 1:40 pm
It seems that a lot of Agilists aren’t interested in solving problems, instead they are only interested in re-packaging and re-selling the same ideas in as many consultation gigs as possible.
For a methodology to be truly useful, I need to be able to apply different pieces in different places at different times, to different degrees. I’ve been able to do this with some aspects of XP/TDD over the years, and I’ve been pretty happy with the results.
But statements like “you wouldn’t have those problems if you started with TDD” hint at an all or nothing philosophy. You must start with XP, and use XP for everything, or you will fail, and cannot legitimately complain about it.
What value is there in a solution if you can’t inject it at any point in the software process? Isn’t that the definition of refactoring: improving the design of *existing* code?
@Sam: So anybody who hasn’t worked on a “real” XP project shouldn’t be able to criticize it? Well, I guess that’s a clever way to eliminate 99.9% of developers from the argument. You and that other 0.01% should have a blast at the TDD tug-off 2006.
#9 by Carsten Saager on June 7, 2006 - 2:24 pm
What is a real and pure Agile or XP-project? Besides some interesting exceptions, their existence is as likely as the tooth-fairy or Santa Clause.
Real-world projects, as I understand Cedric, are where you have to compromise. If your “process” only works if you follow it 100% it is useless, because this will virtually not happen at all.
The point I draw from Agile is to draw back more responsability and responsiveness to the implementation team which is a good thing.
A word (or some more) on tests: I neither like tests as a replacement for documentation nor specification. As “code stays forever” documentary test-cases have a tendency to reflect overcome uses. As specs they are simply too limited and raise the question who will implement them. Nevertheless, tests which show proper use are important as they show the functionality in its purest form and help you stabilizing refactorings and redesigns. Tests that implement use-cases from specs are extremely useful if you work with multiple tiers as they serve as a formalized spec to define and control the interface.
Not testable = useless: LOL – I think our customers would be happy if we scrap 99% of the application: How do you design a test that shows that a collection of objects is in a consistent state after you applied 10000 transactions on it? How do you test foresight? Most applications I know replace existing systems and are implemented incrementally; the early stages don’t need elements that are crucial for the advanced modules. They cannot be tested efficiently from start, but it is possible to implement them – that’s why you want domain experience. Ignoring this experience becomes a death-kiss to the project, bad thing is that you realize this perhaps two years too late. It is not only the correctness of the code, this is only a necessary (if at all) but not a sufficient condition for a project’s success.
#10 by anjan bacchu on June 7, 2006 - 3:07 pm
hi cedric,
a while back, mike spi-lle posted his frustrations on a similar issue — http://www.pyrasun.com/mike/mt/archives/2004/07/10/18.43.16/index.html.
BR,
~A
#11 by anjan bacchu on June 7, 2006 - 3:13 pm
hi cedric,
a while back, mike spi-lle posted his frustrations on a similar issue — http://www.pyrasun.com/mike/mt/archives/2004/07/10/18.43.16/index.html.
BR,
~A
#12 by Don McCaughey on June 7, 2006 - 3:51 pm
I think it’s unfair to say “Very often, Agilists simply forget that their job is to produce software that satisfies customers, not software that meets some golden software engineering scale.” — this is true of all programmers.
As far as Jeff’s presentation goes, I was also very disappointed. I’m working on a large XP project with a dozen engineers, developing an internal app. Given the title of the talk (“Speeding Up Development With TDD”), I was hoping for more practical information, but unfortunately Jeff’s talk was a sermon aimed at converting the uninitiated.
As far as Jeff’s two points that bothered you, they bothered me too. I think they are overly simple.
* Tests are (executable) specs.
One important principle of agile methodologies is to minimize the amount of low-value work your team does. One very common low-value task is preparing detailed specifications and documentation for an internal system with rapidly changing requirements. Spending hundreds of hours to write and maintain _detailed_ specs and docs for a typical business app, which will be read by an audience in the tens of people, is very low-value work. In this case, a high level overview written in a couple of days will provide a much higher value to the customer.
However, if you’re Josh Bloch writing libraries for the JDK, then writing detailed specs and docs for an audience of hundreds of thousands of Java developers _is_ high-value work. If your app is part of a drug or medical device manufacturing line, FDA regulations require you to prepare detailed specs, docs and test plans, so it’s also high-value work in this case.
Very few people like to write documentation, and when they embrace agile, some follow a flawed line of reasoning that goes from “minimize low-value work” to “don’t do detailed documentation” to “your code and tests are your documentation” to “tests are executable specs”.
* If it’s not testable, it’s useless.
I’d state this more like, “if it’s not tested, it’s risky”. Developers write tests to verify intent, prevent regression and enable change. If part of your system doesn’t have test coverage, you don’t have direct knowledge that it works as intended, you may change something that causes it to break and not notice right away, and your system becomes harder to change.
Not writing tests because you have a looming deadline may seem like an acceptable trade-off in the short run if you’re an extraordinary coder whose code always matches their intent (I’m not one), but you leave a bitter legacy for yourself or others who follow when you need to move development forward after the release. Your system regresses more often and the cost of change grows.
If you’ve worked on an app that has logic in SQL and/or JavaScript, but no test coverage for those parts, you’ve experienced this first hand. The point of agile methodologies is to reduce the risk of change. If you have areas of your system that don’t have automated test coverage, you increase the risk of change and you don’t provide the best value to your customer.
When you do have an area of your system that isn’t practicable to test, you do your best to isolate it and manage that risk by limiting how your code interacts with it and doing more manual QA targeted at it, if possible.
That said, an important virtue of developer written tests is parsimony. You should write just enough test code to verify your intent and prevent regression. If you are testing a numerical function (to borrow an example), you don’t need to try and “specify” the formula in your tests by testing every input value, but rather test a few key values to be sure you’re getting correct answers and handling important boundary conditions.
* The Stack example
This is a terrible way to illustrate TDD. Particularly since Jeff ended up using a LinkedList as the internal implementation, so that his final Stack implementation was a nearly empty wrapper around LinkedList. (He did emphasize refactoring the test code, which was the one good point I thought he made.)
I got the feeling that Jeff’s experience with agile development was broad but not very deep — perhaps he’s consulted for a few months at a time with many teams adopting agile methods, but hasn’t worked on any long projects where the team has been committed and skilled users of agile methods.
#13 by Behrang on June 7, 2006 - 4:21 pm
Wow! A very well written article man! Actually I belive the problem with most advocates of a certain way of thought is hasty generalization and over hyped representation of certain facts. In case of Agilists, Instead of saying “your code would be more usable if it is testable and tested”, they say “If it’s not testable, it’s useless” and so on…
So far I have only worked in places that claim to follow RUP. Alas I haven’t seen anything RUP in action so far. Just a bunch of useless outdated UML diagrams… Seems RUP is so expensive that cannot be implemented…
#14 by Behrang on June 7, 2006 - 4:24 pm
Wow! A very well written article man! Actually I belive the problem with most advocates of a certain way of thought is hasty generalization and over hyped representation of certain facts. In case of Agilists, Instead of saying “your code would be more usable if it is testable and tested”, they say “If it’s not testable, it’s useless” and so on…
So far I have only worked in places that claim to follow RUP. Alas I haven’t seen anything RUP in action so far. Just a bunch of useless outdated UML diagrams… Seems RUP is so expensive that cannot be implemented…
#15 by Jason on June 7, 2006 - 5:15 pm
Although I agree with a lot of what you say, I think you are perhaps going too far in the other direction when it comes to the claim that “If it’s not testable, it’s useless”. In particular, quotes like this worry me:
“But the truth is: at times, I don’t do TDD because implementing a feature quickly is more important than a fuzzy feeling.”
This is at best short-sighted, and at worse a complete fallacy. Certainly, you can knock up the feature code quicker than the feature + test code. Does this mean you have the feature quicker? Maybe … but since you haven’t tested it how will you know? So you might decide to test it by hand … but is this really more efficient than writing an automated test? Particularly as you discover problems and need to retest? Even in this short term view, although counter-intuitive, it may be faster to implement the tests properly.
In the long term, it is certainly more efficient to have a proper test suite in place. So even if cutting corners helps you get to the deadline faster, the next deadline will surely suffer. You’ll be busy fixing the bugs from the untested code and will have no confidence in new changes made without repeatable tests. Agile or otherwise, all good development teams understand this.
Pragmatically speaking, there is certainly a threshold where adding more tests will actually start to slow you down. But I think we too easily jump to the conclusion that writing less code in this moment actually saves time, even for short term goals.
#16 by Jason on June 7, 2006 - 5:24 pm
Although I agree with a lot of what you say, I think you are perhaps going too far in the other direction when it comes to the claim that “If it’s not testable, it’s useless”. In particular, quotes like this worry me:
“But the truth is: at times, I don’t do TDD because implementing a feature quickly is more important than a fuzzy feeling.”
This is at best short-sighted, and at worse a complete fallacy. Certainly, you can knock up the feature code quicker than the feature + test code. Does this mean you have the feature quicker? Maybe … but since you haven’t tested it how will you know? So you might decide to test it by hand … but is this really more efficient than writing an automated test? Particularly as you discover problems and need to retest? Even in this short term view, although counter-intuitive, it may be faster to implement the tests properly.
In the long term, it is certainly more efficient to have a proper test suite in place. So even if cutting corners helps you get to the deadline faster, the next deadline will surely suffer. You’ll be busy fixing the bugs from the untested code and will have no confidence in new changes made without repeatable tests. Agile or otherwise, all good development teams understand this.
Pragmatically speaking, there is certainly a threshold where adding more tests will actually start to slow you down. But I think we too easily jump to the conclusion that writing less code in this moment actually saves time, even for short term goals.
#17 by Chui on June 7, 2006 - 7:07 pm
Specs are more than executable tests.
In any deployed codebase, even unspecified behavior form de facto specification. Witness how MS software breaks whenever they try to change an internal implementation, or how people rely on certain buggy features to perform their work.
#18 by Hani Suleiman on June 7, 2006 - 8:44 pm
it’s probably no coincedence that many/most agilists (at least,that I know of, they don’t tend to be a closeted species) are consultant types; an environment where maintanability is not an issue, and the fact that it takes a long time to get acquainted is a *good* thing.
My beef with TDD is that it just doesn’t scale to a real world team. I don’t mean a particularly large one, I mean one with average developers.
Writing tests first is harder than writing to some spec that someone smarter than you has come up, and most people will botch it up. That relegates it to the realm of ‘neat, but not particularly useful, nor will it catch on in the mainstream’ things, to me.
#19 by Mishkin Berteig on June 7, 2006 - 9:19 pm
You said: “Let’s talk about TDD for code that interacts with clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place”
Okay!
Personal Example #1: I used TDD for one part of a very large enterprise framework (don’t know how many kloc, but definitely > 500). The rest of the framework was not built using TDD. My part was the guaranteed delivery messaging system (a JMS implementation to be specific). It was written in Java and used in a large financial institution. When I was finished, it went through QA and Pre-Production testing with zero defects found. It ran in a 2000-server production environment for 2 years before any defects were found. The defect found was a very very obscure deadlocking condition (oh yes, not only was the system a messaging system, but it was also locally multi-threaded).
Example #2: I used TDD with a small team of 4 developers to build a high-volume multi-processor database-intensive data processing system. The system was smaller in terms of kloc (maybe 20), but again, used in a very serious financial institution. This project used C#/.NET and relied on several non-TTD-friendly external libraries. In this particular institution, this was the first project _ever_ that was delivered on time. TDD and the resulting high quality had a lot to do with it. Although we did find one nasty bug in QA that haunted us a little, it was again a threading/multi-processor related bug. All the business logic was flawless.
Your objections to TDD and the XP approach also fail to acknowledge the incredible value provided by the management oriented practices of agile methods. Don’t forget that XP isn’t just TDD and pairing. There is also iterative deliver, adaptive planning, user stories, collective code ownership, customer collaboration, etc. etc. etc.
I happen to be one of those “consultant type” agilists. I focus more on the management and team aspects rather than the techincal aspects (although I come from a techie background). To me, TDD is, as you say, really easy to describe and for anyone with at least one functional neuron, easy to learn. The hard part is doing it consistently and as a discipline.
Thought-provoking article. Thanks.
Oh, and if you’re interested, I write about the agile management stuff at http://www.agileadvice.com/.
#20 by Mishkin Berteig on June 7, 2006 - 9:21 pm
You said: “Let’s talk about TDD for code that interacts with clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place”
Okay!
Personal Example #1: I used TDD for one part of a very large enterprise framework (don’t know how many kloc, but definitely > 500). The rest of the framework was not built using TDD. My part was the guaranteed delivery messaging system (a JMS implementation to be specific). It was written in Java and used in a large financial institution. When I was finished, it went through QA and Pre-Production testing with zero defects found. It ran in a 2000-server production environment for 2 years before any defects were found. The defect found was a very very obscure deadlocking condition (oh yes, not only was the system a messaging system, but it was also locally multi-threaded).
Example #2: I used TDD with a small team of 4 developers to build a high-volume multi-processor database-intensive data processing system. The system was smaller in terms of kloc (maybe 20), but again, used in a very serious financial institution. This project used C#/.NET and relied on several non-TTD-friendly external libraries. In this particular institution, this was the first project _ever_ that was delivered on time. TDD and the resulting high quality had a lot to do with it. Although we did find one nasty bug in QA that haunted us a little, it was again a threading/multi-processor related bug. All the business logic was flawless.
Your objections to TDD and the XP approach also fail to acknowledge the incredible value provided by the management oriented practices of agile methods. Don’t forget that XP isn’t just TDD and pairing. There is also iterative deliver, adaptive planning, user stories, collective code ownership, customer collaboration, etc. etc. etc.
I happen to be one of those “consultant type” agilists. I focus more on the management and team aspects rather than the techincal aspects (although I come from a techie background). To me, TDD is, as you say, really easy to describe and for anyone with at least one functional neuron, easy to learn. The hard part is doing it consistently and as a discipline.
Thought-provoking article. Thanks.
Oh, and if you’re interested, I write about the agile management stuff at http://www.agileadvice.com/.
#21 by Mishkin Berteig on June 7, 2006 - 9:22 pm
You said: “Let’s talk about TDD for code that interacts with clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place”
Okay!
Personal Example #1: I used TDD for one part of a very large enterprise framework (don’t know how many kloc, but definitely > 500). The rest of the framework was not built using TDD. My part was the guaranteed delivery messaging system (a JMS implementation to be specific). It was written in Java and used in a large financial institution. When I was finished, it went through QA and Pre-Production testing with zero defects found. It ran in a 2000-server production environment for 2 years before any defects were found. The defect found was a very very obscure deadlocking condition (oh yes, not only was the system a messaging system, but it was also locally multi-threaded).
Example #2: I used TDD with a small team of 4 developers to build a high-volume multi-processor database-intensive data processing system. The system was smaller in terms of kloc (maybe 20), but again, used in a very serious financial institution. This project used C#/.NET and relied on several non-TTD-friendly external libraries. In this particular institution, this was the first project _ever_ that was delivered on time. TDD and the resulting high quality had a lot to do with it. Although we did find one nasty bug in QA that haunted us a little, it was again a threading/multi-processor related bug. All the business logic was flawless.
Your objections to TDD and the XP approach also fail to acknowledge the incredible value provided by the management oriented practices of agile methods. Don’t forget that XP isn’t just TDD and pairing. There is also iterative deliver, adaptive planning, user stories, collective code ownership, customer collaboration, etc. etc. etc.
I happen to be one of those “consultant type” agilists. I focus more on the management and team aspects rather than the techincal aspects (although I come from a techie background). To me, TDD is, as you say, really easy to describe and for anyone with at least one functional neuron, easy to learn. The hard part is doing it consistently and as a discipline.
Thought-provoking article. Thanks.
Oh, and if you’re interested, I write about the agile management stuff at http://www.agileadvice.com/.
#22 by Kamal on June 7, 2006 - 10:35 pm
I think this article was written in haste, provoked by a single (few) incident(s). Coming from people like you it will do a lot of harm.
It is wrong to generalize the attitude of a few and blame the whole agile crowd.
Every technology / methodology has its own share of zealots. People who think they are doing it but arent really doing it right.
I have been attracted to the agile practices mainly because of the pragmatic approach and the emphasis on delivering value to the customers. Over the past few years I have spent a lot of time reading what people like Dave Thomas, Andy Hunt, Robert Martin, Craig Larman, Ron Jeffries, Martin Fowler, Michael Feathers have written. The more I read and more I think about it I am convinced that for projects to be successful they need to adopt some of these practices.
There are people who are making an attempt to go beyond the toy problems and show you how to apply these practices. One good example is this book
Working Effectively with Legacy Code
http://www.amazon.com/gp/product/0131177052/104-6025377-3148732?v=glance&n=283155
Most of the popular open source projects are proof that you can deliver successful software without specs. I have never come across a requirement spec or design spec for things like hibernate, springframework. I have even come across a number of cases where people are being refered to tests to understand how something works where there was inadequate user documentation.
Hope you will reflect on it a bit more.
#23 by Tim Vernum on June 8, 2006 - 12:51 am
> but is this really more efficient than writing an automated test?
TDD != Automated Testing
TDD implies writing the tests first and then the code. (Often it implies 1 test, then a small amount of code, then another test, then more code…)
It may or may not be a good way to get well designed code. It’s definately not the _only_ way to get well designed code. And it’s not the _best_ way in _every_ situation.
The software game is about products, not processes. Good processes help produce good products. Automated testing is a good process.
That’s about as strict a set of statements as we can make.
If you want to say that “the quality of the processes are the sole determinant of the quality of the product” or “automated testing is the best process” then you’re engaging in unnecesary (and dishonest) hyperbole.
#24 by Lee Meador on June 8, 2006 - 8:29 am
Saying “tests can have bugs” is misleading and not useful to the issue of whether tests can be specs. Any sort of specs can have mistakes. A bug is just what we call a mistake in the code.
You could specify that the name be shown as “Last, First” but if that isn’t correct …
#25 by Daniel Serodio on June 8, 2006 - 9:46 am
IMHO, that’s what the “Extreme” in “Extreme Programming” means: shocking people. XP is not really extreme, but its proponents are always trying to shock their audience.
Shocking the audience can have 2 opposite effects: either they’ll think “wow, I’d never thought about this before, let’s listen to what this guy is saying”, or “this guy is full of s***, let’s get back to work”.
I really like TDD, but if I heard “if it’s not testable, it’s useless”, I’d go back to “the real world” and work.
I prefer the motto “if it’s not testable, it’s detestable”.
And this “if didn’t work for you it’s because you didn’t follow it properly” excuse is really lame and old.
#26 by sammy on June 8, 2006 - 11:39 am
I’m going to speak out in defense of Sam, the first commenter, and not just because we share a first name.
In your post, you lambast Agile developers for using “useless” examples in classroom settings like “let’s code a Stack.”
When Sam claimed to have worked on a project of signifigant size and that he used XP successfully on it, mmatt criticized him for “putting himself on a pedestal.”
So in other words, demonstrating that it works in a small case is useless, while claiming that it works in a large case is bragging. With standards like that, it’s no wonder noone can prove to you guys that agile software methods are effective.
(Disclaimer: I don’t even particularly like “agile methodology” beyond the continuous integration and rigorous testing aspects.)
#27 by sammy on June 8, 2006 - 11:40 am
I’m going to speak out in defense of Sam, the first commenter, and not just because we share a first name.
In your post, you lambast Agile developers for using “useless” examples in classroom settings like “let’s code a Stack.”
When Sam claimed to have worked on a project of signifigant size and that he used XP successfully on it, mmatt criticized him for “putting himself on a pedestal.”
So in other words, demonstrating that it works in a small case is useless, while claiming that it works in a large case is bragging. With standards like that, it’s no wonder noone can prove to you guys that agile software methods are effective.
(Disclaimer: I don’t even particularly like “agile methodology” beyond the continuous integration and rigorous testing aspects.)
#28 by Daniel Serodio on June 8, 2006 - 1:01 pm
The point is that Sam failed to mention any number about the “*real* project”. How many people/classes/LoCs/months ? Anything?
Without numbers, we can only assume that he is bragging.
#29 by Marcus Widerberg on June 8, 2006 - 4:39 pm
This was so stupid it hurts! I want my minutes back!
#30 by Thiago Arrais on June 8, 2006 - 7:42 pm
Tests certainly aren’t specs, but they are executable. That’s what makes them so useful, they provide very fast feedback. And that’s the whole point behind this agile thing: reducing the time for feedback by means of highly iterative practices.
#31 by Bruce T on June 8, 2006 - 10:09 pm
Nice blog in some ways, but paints with much too broad a brush. See my comments at [paddle like hell](http://blog.rapidred.com/articles/2006/06/09/people-who-generalize-suck.)
#32 by Anonymous on June 9, 2006 - 6:06 am
First off, I respect you for standing up to the agile religion, it’s like scientology or something… Personally, I’ve been fearful of speaking strongly against agile, even though that means watching US engineering continue to spiral down the toilet. It’s funny because we have 50 years of engineering software experience that shows us how most of these agile practices don’t effectively work.
There are a couple things, first tests and TDD are nice but there are costs and as the project grows the whole process has to slow down because tests balloon the code base (it’s usually worth while) but I’ve seen more than a few cases where these simple quick deep changes resulted in retooling hundreds of testcases. The more test coverage you get, ironically, the less quickly you can do certain things but you end up with more trust in those things.
Also, there are problem spaces where agile just doesn’t work. Most agilists tend to be web developers or consultants. If a web page is a “work item” then agile is beautiful or if you don’t plan on being there when/should they get to the finish line then agile might work for you. There are no embedded agilists. There are no system agilists. You don’t evolve a design like TCP’s. So long as your goal is to simply use technology and not provide any, agile might work just pick Rails or Jboss off the shelf and build a website on top of it and iterate on the look and feel and you’ll be super agile. Does a bank want monthly releases? Secondly, when you sell to customers like banks the weight of releases increases, you simply cannot “reset them” or screw up a release after they are deployed. I know that’s a problem that should be addressed but it is a constantly moving target in many cases, it’s not a fix it and forget it problem, not unless you’re making a little piece of software.
#33 by Dave C on June 9, 2006 - 6:29 am
Ah…the backlash.
As a long-time “real world” agile practitioner (not as a consultant), let me chime in.
Practices are not the same as principles. The definition of “Agile” is not TDD + CI + PairProgramming + IndexCards = Success
Software development is ENTIRELY about the people working on a project, and their ability to adapt to what’s happening. In my mind, this adaptation is the core of agility.
In my own experience as a developer and dev manager, I was unable to use the exact same Agile approach on different projects. I had to tune it for the situation, sometimes adding practices, sometimes eliminating them.
So to me this means that anyone claiming that a single set of practices is the One True Way ™ is probably selling something or just inexperienced.
Having said that, there are practices that seem to have universal value, agile or not. I happen to think automated tests are a good thing, whether they are done first or last.
From a project management standpoint, short iterations and frequent delivery to the customer have been invaluable to me.
Your mileage may vary, but let’s not paint everyone who believes in Agile approaches as a lunatic. But I understand the source of this rant.
#34 by Dave C on June 9, 2006 - 6:30 am
Ah…the backlash.
As a long-time “real world” agile practitioner (not as a consultant), let me chime in.
Practices are not the same as principles. The definition of “Agile” is not TDD + CI + PairProgramming + IndexCards = Success
Software development is ENTIRELY about the people working on a project, and their ability to adapt to what’s happening. In my mind, this adaptation is the core of agility.
In my own experience as a developer and dev manager, I was unable to use the exact same Agile approach on different projects. I had to tune it for the situation, sometimes adding practices, sometimes eliminating them.
So to me this means that anyone claiming that a single set of practices is the One True Way ™ is probably selling something or just inexperienced.
Having said that, there are practices that seem to have universal value, agile or not. I happen to think automated tests are a good thing, whether they are done first or last.
From a project management standpoint, short iterations and frequent delivery to the customer have been invaluable to me.
Your mileage may vary, but let’s not paint everyone who believes in Agile approaches as a lunatic. But I understand the source of this rant.
#35 by Emmanuel Pirsch on June 9, 2006 - 7:02 am
Here is my take on the matter : http://epirsch.blogspot.com/2006/06/agility-is-about-customer.html
#36 by Julio on June 9, 2006 - 7:23 am
> They offer you all these nice ideas such as Test-Driven Development and Pair Programming but they never — ever — disclose the risks and the downsides. To them, Agility is a silver bullet that is applicable in all cases with no compromises.
You couldn’t have more dishonest (or ignorant of agile communities)
#37 by Chris on June 9, 2006 - 7:46 am
The “Agile Thugz” comment is the funniest thing I’ve read in a while. Fuh’real Fuh’real, word to my muva.
#38 by Kane on June 9, 2006 - 10:18 am
You pose a good problem when you asked: “How do you specify an exponential function with a test?” This doesn’t directly answer you question, but I thought your audience might be interested:
http://www.extremeperl.org/bk/test-driven-design
#39 by Muk on June 9, 2006 - 11:09 am
Well – this has been bothering me for sometime so I think I will come clean. There are things about TDD that lend themselves well to certain forms of development like POJO/framework development. Java development that depend on Enterprise infrastructure do not lend themselves well. So at that point do what is pragmatic and test what pieces you can.
Documenting code certainly has merits and the flip side is true as well – it is more likely to get out of whack. Larger software teams will not
lend themselves to agile and if “time to market”
is critical – which is the case with most professional companies you cannot TDD fully.
So I see Cedric’s points. I have yet to follow any methodology fully on a project without improvising and see it work. You see, evaluate, adopt what works well and move on…
#40 by Anonymous on June 9, 2006 - 2:54 pm
I’m not going to bother to defend my post because it’s pointless. There will always be some BS excuse about how my particular situation was unique in some way that made it possible.
I’m only posting to say that most people I run into don’t really believe in Agile development. They may think there’s a few good things in there to learn from but overall it’s just “pie in the sky” bullshit. Consequently, there are very, very, very few Agile projects actually in existence. As a consultant I get to work around at a lot of different companies. And even with that going for me, the odds of me getting to work on more Agile projects as time goes on is close to zero.
So don’t worry Agile haters. It’s more of a neat idea. You’re all perfectly safe in your little worlds that you guard so carefully. In the end it doesn’t matter to me. I get paid the same amount whether the processes I use are retarded or brilliant.
#41 by Muthu Ramadoss on June 9, 2006 - 9:59 pm
You CANNOT do AGILE in India. Period.
If you want to, you need to join Thoughtworks, India.
I am saying the above considering the practical realities of Software Development in India.
Muthu Ramadoss.
http://groups.google.com/group/etoe
#42 by Muthu Ramadoss on June 9, 2006 - 10:00 pm
You CANNOT do AGILE in India. Period.
If you want to, you need to join Thoughtworks, India.
I am saying the above considering the practical realities of Software Development in India.
Muthu Ramadoss.
http://groups.google.com/group/etoe
#43 by Anonymous on June 10, 2006 - 8:33 am
I think if you use Scrum you do alleviate some of your concerns about the spec. Actually the spec quality can increase since the product team can continue refining the product backlog during the sprint (the items that are not contained in the sprint backlog).
#44 by Anonymous on June 10, 2006 - 8:33 am
I think if you use Scrum you do alleviate some of your concerns about the spec. Actually the spec quality can increase since the product team can continue refining the product backlog during the sprint (the items that are not contained in the sprint backlog).
#45 by Anurag Shrivastava on June 11, 2006 - 11:34 pm
You can do Agile in North India for sure. Xebia, a subsidiary of Xebia IT Architects B.V. in Holland, is a software house specializing in Agile and Iterative methodologies. We have a working offshore Agile Development Practice. I will be interested in getting touch with people who would like to share their Agile experience on Java/J2EE based projects.
Anurag
#46 by Anurag Shrivastava on June 11, 2006 - 11:35 pm
You can do Agile in North India for sure. Xebia, a subsidiary of Xebia IT Architects B.V. in Holland, is a software house specializing in Agile and Iterative methodologies. We have a working offshore Agile Development Practice. I will be interested in getting in touch with the people who would like to share their Agile experience on Java/J2EE based projects.
Anurag
#47 by Robert C. Martin on June 12, 2006 - 6:02 am
I have responded to this on:
http://butunclebob.com/ArticleS.UncleBob.AgilePeopleStillDontGetIt
#48 by Michael Feathers on June 12, 2006 - 6:37 am
Cedric said: >>>To be honest, I am becoming quite suspicious of Agile practices for that reason: all the presentations I have attended and books that I have read are always using toy implementations as examples. Stack, List, Money, Bowling… enough already! Let’s talk about TDD for code that interacts with clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place (and: yes, I read Michael Feathers’ book, it has some good and some bad, but it’s not germane to Java and TDD so I won’t expand on it here).<<<
Are you serious? Well over half of the examples are in Java, and yes it is TDD: you get the test around the code you want to change first.
Re presentations about TDD. I share your concern, but frankly, I do exactly the same thing that Jeff did most of the time. Whenever you’re presenting something you have to figure out what to leave out so that people aren’t swamped with irrelevant detail; so that they can see the forest for the trees. I call it ‘pedagogical challenge’ and it’s a pain. You’ll always find people who won’t think it’s real to them unless the example resonates with some aspect of their personal experience, but you try to reach who you can and often that means picking something non domain-specific so that everyone can engage.
The fact is, TDD is pretty independent of context. Name a class that is supposed to work in your “clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place.” If we paired together, we’d do the same thing that we would do with bowling or a stack. They are the same steps, really. We’d figure out what logic we need, and we’d develop it test first, independently of all the other crap, in a test harness. It takes work to get people to get that.. that it’s all about building things independently.
‘Working Effectively in Legacy Code’ can really be seen as a cautionary tale. That’s what you have to go through when your code wasn’t designed for test. I’m not one of those who tell people “if you used TDD you wouldn’t be in this mess” but I do imply it and I help them discover it. In fact, my favorite coaching technique right now is to help groups start to use TDD on the easy bits, the new features that look like they can be independent and they discover how it makes development easier. Then we go in and try to change existing code.. we notice that we’re not really sure of our changes like we were in the TDDed code. But we want to be sure so we go in and do all sorts of tricky work to get tests in place. It is tricky work and it is a pain in the ass. And, that’s usually enough to sell people. If you’ve worked in a good TDDed code base and you do the latter work often enough, you realize just how insane it is not to TDD from the beginning and it is hard to bite your tongue and not give that message to people who aren’t prepared for it. But giving them the experience does help.
#49 by Brad Appleton on June 12, 2006 - 8:21 am
I happen to be a person who works in a large telecom company on large projects that has been adopting and adapting agile methods over the past 5 years with stellar improvements in quality, productivity and cycle-time as the fruits of our labors. Our organization consists of many thousands of people across several divisions, each of which works on one or more systems with hundreds of engineers, tens of millions of lines of code, with subsystems in the 1M+ line of code range, and components in the 100K+ line of code range.
My first comment would be that Cedric’s post seems specific to XP, rather than all (or even most) agile methods. Take a look at Feature-Driven Development or FDD (http://www.featuredrivendevelopment.org/), which is quite different from XP and has none of the things that Cedric is complaining about.
Secondly, I can say that much of what Cedric says rings both true *and* false in my organizations adapting of Agile methods to our large projects. Our tailored Agile method combines elements of XP, Scrum, FDD, and traditional systems engineering. On the surface it bears the most resemblance to a mix of systems engineering and XP.
We adopted practices like TDD, CI, Refactoring, Pairing, “Simple” Design, and a few others. However we also still do plenty of requirements documentation, we use formal tracking systems for managing change-requests and problem-reports, we make use of model-driven development, and we do comment our code and write high-level architecture/design docs (but much more lightweight than before — trying to stick to high-level information that spans more than one file/class of a component).
Agile doption in the company has been progressing slowly but surely. We definitely run into skeptics; The most compelling persuasion for our skeptics is successful real-world results with measurable improvement in each area claimed. It works most effectively when the skeptics participate in the success
#50 by Jeff Langr on June 12, 2006 - 4:36 pm
Greetings Cedric,
It’s a long blog posting, and I just wanted to comment on a few errors you made.
1. “tests are specs” is a mindset, a generalization, and I made this point clear when Bloch (I think) challenged it. In fact, that was the title of the slide as presented: “Mindsets”. It helps to recognize the idea of treating these as specs, because most people doing TDD do a poor job of coding them to be readable.
2. “If it’s not testable, it’s useless” was on the same slide, about mindsets. Both of these are attitudes. The more people move in the direction of these attitudes, the better. There are always exceptions.
I even said that these were things that helped me think in terms of how I want to approach TDD.
3. No, “we” do *not* get TDD. Most people don’t. Most developers doing it based on reading the 30 second blurb do it poorly. Then they grouse, like you do, that it’s not all testable. Usually, the inability to test close to 95% or more of your app suggests you have a bad design.
4. “How about: if we pop an empty stack, we get an exception”
“Mmh, no, let’s not do that”
You did not hear my response to this. I suggested that popping from the empty stack doesn’t have anything to do with “testCreate” for a stack. Which it doesn’t–it’s separate behavior. It’s on the tape, if you wanted to watch it again.
5. “Well, if you had started with TDD in the first place, you wouldn’t be having this problem today”.
I even admitted this was a glib answer and suggested why.
I’m pretty disgusted with the way you’ve misrepresented my talk. I think you’re the one who’s being dishonest, Cedric.
—
I was invited to google to speak with the understanding that the audience was interested in an introductory level discussion. Hence the simplistic example, and also hence my emphasis on technique. My apologies that it didn’t get communicated that way. Frankly, I find the fact that you suggest that I was being dishonest slanderous and ultimately just disappointing.
I don’t work on open source. I work with real customers, most of them Fortune 500, who have far worse code than they need to have. I’m currently working with one customer who took 190,000 lines of code 9 sprints ago, and reduced it to 60,000 since then, all along adding new features. They did this by virtue of doing TDD the “right” way. Not by making excuses about why they couldn’t test it all.
I’ve been excited about working with companies and doing TDD, because I’ve seen it work when done well, and I’ve seen the dramatic decreases in cost. So here’s my advice to you, Cedric, and to Google: you can stay on your high horse and post misleading comments. I’m just going to keep doing what I know can work well.