I just attended Test-Driven Development presentation which represents everything that is wrong about the way Agile advocates are trying to evangelize their practices. I don’t have anything against the presenter in particular, but it’s really time for Agilists to rethink the way they communicate with the real world.
Here are a few comments on his presentation.
One of the first slides that deeply troubled me claimed the following:
- Tests are (executable) specs.
- If it’s not testable, it’s useless.
First of all, tests are not specs. Not even close. Somebody in the audience was quick to give a counter-example to this absurd claim by using a numeric example ("how do you specify an exponentiation function with a test?") but my objection to this claim is much broader than that. Relying on tests as a design specification is lazy and unprofessional because you are only testing a very small portion of the solution space of your application (and of course, your tests can have bugs). Tests also fall extremely short of having the expressiveness needed to articulate the subtle shades that a real specification need to cover to be effective.
This claim is part of a broader and more disturbing general Agilist attitude that is usually articulated like "Your code is your spec", along with some of its ridiculous corollaries such as "Documentation gets out of date, code never does".
Anyone who claims this has never worked on a real-world project. And I’m setting the bar fairly low for such a project: more than five developers and more than 50,000 lines of code. Try to bring on board new developers on this project and see how fast they come up to speed if all they have to understand the code base is… well, just code. And tests.
I am currently getting acquainted with a brand new project that is not even very big, and while I understand Java fairly well, there is no doubt in my mind that for ten minutes I spend trying to understand how a certain part of the application works, a five-line comment would have given me this knowledge in ten seconds.
The second claim, "If it’s not testable, it’s useless" is equally ludicrous and a guarantee that at this point, the audience you are talking to is already looking at you as a crackpot.
Software is shipped with untested parts every day, and just because it’s not entirely tested doesn’t mean it’s bad software or that the untested parts are "useless".
Agilists just don’t understand the meaning of calculated risk.
Early in the development cycle, it’s perfectly acceptable to go for a policy of "zero bugs" and "100% tests". But as the deadline looms, these choices need to be reconsidered all the time and evaluated while keeping a close eye of the final goal. Very often, Agilists simply forget that their job is to produce software that satisfies customers, not software that meets some golden software engineering scale.
Anyway, let’s go back to the presentation, which then proceeded with the implementation of a Stack class with TDD. Before spending thirty minutes on a live demo of the implementation of a Stack class (are you impressed yet?), the presenter warned the increasingly impatient audience that they should "not pay too much attention to the Stack example itself but to the technique".
And that’s exactly the wrong thing to do.
Look, we "get" TDD. We understand it. Frankly, it takes all of ten minutes to explain Test-Driven Development to a developer who’s never heard of it: "Write a test that fails and doesn’t compile. Make it compile. Then make it pass. Repeat".
The hard part is applying it to the real world, and showing the implementation of a Stack will soon have everyone leave the room with the thought "Cute, but useless. Now let’s go back to work".
It was even worse than that, actually: The presenter kept taking suggestions from the crowd but he declined all those that didn’t fit in the neat script that he had in hands at all times. These suggestions were good, by the way:
"What should we test now?"
"How about: if we pop an empty stack, we get an exception"
<a bit embarrassed> "Mmh, no, let’s not do that" <hand waving, look down at notes and proceeds while happily ignoring the other raised hands>
To be honest, I am becoming quite suspicious of Agile practices for that reason: all the presentations I have attended and books that I have read are always using toy implementations as examples. Stack, List, Money, Bowling… enough already! Let’s talk about TDD for code that interacts with clustered databases on laggy connections built on 500,000 lines of code that was never designed to be tested in the first place (and: yes, I read Michael Feathers’ book, it has some good and some bad, but it’s not germane to Java and TDD so I won’t expand on it here).
And please, avoid smug and useless answers such as:
"A lot of the classes I have to test are hard to isolate, do you have any advice regarding mocks?"
"Well, if you had started with TDD in the first place, you wouldn’t be having this problem today".
Fundamentally, I am disturbed by the Agilists’ dishonesty when it comes to presenting their arguments. They offer you all these nice ideas such as Test-Driven Development and Pair Programming but they never — ever — disclose the risks and the downsides. To them, Agility is a silver bullet that is applicable in all cases with no compromises.
The truth is that these practices come at a price, and for a lot of organizations, the price gets high very quickly. Agile development will never go far if its proponents keep ignoring these organizations and make condescending comments to its members.
I like Test-Driven Development. I really do, and I’m fortunate enough to work on a project that lets me use TDD most of the time. But the truth is: at times, I don’t do TDD because implementing a feature quickly is more important than a fuzzy feeling. And I’m also aware that TestNG is an open source project with less than five developers, all of them on the bleeding edge and aware of the latest advances in software engineering.
And this is my main beef with Agilists: I strongly suspect that most of them are spending their time on open source projects with like-minded fellows, but none of them have any experience what companies whose survival depends on shipping software have to go through to organize huge code bases growing thousands of lines of code every day under the combined push of hundreds of a developers, all with their personal background, education and bias.
So here is my advice to Agilists: get real now, or you will become irrelevant soon.
Update: four years after I posted this article, reddit picked it up.
#1 by Scott on December 19, 2008 - 1:04 pm
Quote
‘Whereof one cannot speak, thereof one must be silent’.
So – from experience on software projects with hundreds of thousand of lines of code accross multiple technologies and business units in a fortune 100…
All activities in the SDLC have a cost/benefit. One should assess the cost/benefit of each activity and try to optimize it.
Though we are not religious about it, we keep trying to improve what we do a little bit at a time. Agile principles, applied judiciously help us to improve the cost/benefit ratio.
I suggest that there is enough evidence that one should try it in a low-risk environment with willing participants. If it works, keep it. If it doesn’t then figure out what you could have done differently and fix it next time.
Of course that last sentence should be applied regardless of methodology.
Large organizations with history tend to have lots of politics. If the message sounds radical or will clearly result in organizational change, it’s going to be hard to gain support.
I’m extremely surprised to see such passionate discourse against Agile come from within a company such as Google. I’d always thought them to be among the more open-minded corporate cultures.
#2 by Eric S on August 5, 2010 - 11:02 am
Quote
The problem is that Agile has become a dogma. Dogma shuts down thinking and discussion. You can no longer discuss the trade offs of benefit vs cost with regard to applying any given Agile practive in a given situation with Agilists because they will not admit such trade offs can ever weigh against an item of dogma. Agile principles have value, applied intelligently and considered in relation to each situation. Having this word “Agile” to encompass these principles has outlived it’s value.
I’d be happy to see it die, but it will not any time soon, because it has become entrenched in many organizations and many people make their livelihood from it. It will continue to poison software development culture until it is eclipsed by the next buzzword dogma.
#3 by Andrew Darnell on August 16, 2010 - 7:06 am
Quote
Arghhh…
I have seen many people take a perfectly good code base and then start ‘sprinting for the line’. The line moves, and they continue sprinting, and they sprint some more… It is getting harder because the ground is changing under peoples feet and Bill over there re-factored a class which broke one of Betty’s assumptions which broke the build for a couple of days before it was tracked down, and in the meantime fifty other change lists went in and now even though the original problem is fixed the build is consistently broken. Never mind some one else will fix it gotta get the new functionality for the sales demo done by tomorrow…
When you look closely, people are spending lots of time debugging rather than coding. The other behaviour I have seen is people avoiding particular areas of code because they seem to be fragile.
The culture shift has been accomplished…
We have all seen projects like this, where a shining tower of simplicity and purity of vision has been changed into a world of intellectual poverty and make do, worthy of the favelas of Rio de Janeiro.
The usual outcome is to then completely re-engineer the next version with a new team because the previous team left the company demoralised, hopefully for happier projects.
It always surprises me how little discipline needs to slip before chaos reigns, and how much chaos bright people can actually manage and put up with before succumbing to the inevitable…
Good agile practice is high discipline, not low discipline…
Dogma is bad, but we have to learn from experience…
#4 by Rich on August 21, 2010 - 12:00 pm
Quote
“Dogma is bad” mmmmm irony
#5 by Giel van Schijndel on August 21, 2010 - 1:51 pm
Quote
First regarding how you dispute the second claim, “If it’s not testable, it’s useless”. While I agree that this guarantees you’re audience to think of you as a crackpot, you dispute it by stating that untested software isn’t (necessarily) bad. The statement was about it being test*able* though, i.e. if it cannot be tested it’s useless. For that I can give a very simple example however: development and maintenance of drivers and embedded software, i.e. often enough you write code for hardware you don’t even have access to so the only test you can perform is “does it compile?” and deal with the bug reports when they come in.
As for you associating Agilists’ badness with open source projects. I’m most definitely not an Agilist (much more an anarchist/non-conventionalist, i.e. I question *every* method I’m expected to use), but about 60~70% of my software development is on open source projects (used to be 100%), and those projects I work on (usually) are of much higher quality than the closed source bases I work with.
Heck, the largest closed source base I work with was developed with some of the aspects of Agile development I despise most; the code *is* the spec and documentation and testing being more important than performing a code review of (critical parts of) the code every now and then. Basically all projects I worked on where “Agile development” was claimed to be used the reality was: no consistent method whatsoever was used, i.e. the used practices were *completely* ad-hoc. That’s my most significant problem with Agile development really: it deteriorates way too easily into ad-hoc development.
#6 by Christopher Dunn on August 21, 2010 - 3:20 pm
Quote
Testing exp() is easy: Test at random points against a known correct implementation, in addition to a few obvious points and any points that have failed in the past. In other words, most of the test changes every time you run it. That can provide whatever level of certainty you desire.
Minor quibbles aside, I do agree with the author here, basically. A test is *not* a spec. However, there is no spec if there are no acceptance tests. It’s too easy for customers to keep their requests unspecific, even with reams of English prose and myriad diagrams.
I do not believe that TDD advocates really are dogmatic. They want the center of opinion to move toward writing tests first. Where I’ve worked, the level of testing has not sufficed, and excuses are rampant. Writing tests in advance is a practical way to ensure that tests are actually written, kind of like obtaining a warrant before breaking into somebody’s house.
#7 by Eduardo on August 21, 2010 - 3:23 pm
Quote
I’ve worked on 50+ people agile teams (20+ developers) as well as 50+ people waterfall teams, and you’re simply wrong.
I specially like your comment about “Having a 10 line paragraph would have helped a lot when I started looking at the code”. I heard that a million times from people joining the team. And you know what? After they worked on the project for a few weeks/months, if you asked them, they would say that having those comments would have been a mistake and a waste.
Your problem seems to be that you just experienced “but-am” agile projects. “We’re doing Agile, but..am.. we don’t really to TDD. We’re doing Agile but..am.. we don’t do pair programming”. Or the worse one “We’re doing agile, but we just do scrum, no xp practices”.
There’s a reason why the XP has a lot of practices. All of them bring something to the table to make the other practices work better. So, for example, when people say “its best not to have comments in the code”, that means that, when you join an Agile team, you’ll be pair programming with an experienced developer, and the explanation you want to read will really be told to you by that person. That contact with the developer will be a lot more valuable than 10 lines of text that will (yes, they will, I’ve seen this a million times) be old and wrong.
There are good and bad presenters for pretty much any topic in the world. I would suggest you look for some better ones instead of taking conclusions based on a couple of people you didn’t like, and then complain if you still want to. A great conference to see people like that will happen in Chicago: http://scna.softwarecraftsmanship.org/ . Google for “Uncle Bob presentation” and watch a couple of videos.
Good luck learning more, or good luck going to the “good” old waterfall.
#8 by Cale Gibbard on August 21, 2010 - 4:27 pm
Quote
I’m not particularly an advocate of TDD, but here’s how I might specify one sort of exponentiation function (written infix as ^) using the QuickCheck testing library in Haskell:
quickCheck $ \x -> x^0 == 1
quickCheck $ \x -> x^1 == x
quickCheck $ \x (NonNegative a) (NonNegative b) -> x^(a + b) == x^a * x^b
Here I’m assuming that our exponentiation function wants the exponents to be nonnegative integers. Depending on which definition of exponentiation we were really trying to capture, variations or extensions of those properties might be in order, but I think that’s a decent start. It completely characterises exponentiation for non-negative integer exponents.
Exponentiation is an easy example because there are lots of well-known algebraic laws you could test. Of course, running the tests in this case will never conclusively prove that your implementation is correct, only catch a decent portion of incorrect implementations a decent portion of the time. Still, writing some tests is often a good excuse for thinking about the algebraic properties that your code ought to satisfy, even if you’re too lazy to write down any proofs.
Libraries with well-thought-out algebraic properties in my experience tend to be the ones which are nicest to actually use, and which scale nicely down to small problems and up to large ones, or from the problems they were originally designed for to new applications.
While I don’t tend to write tests first (and often *gasp* don’t write tests at all), I do spend a lot of time thinking about the relationships and properties that my code needs to satisfy. (Usually much more time than it ends up taking to write the code.)
#9 by Paddy3118 on August 21, 2010 - 7:25 pm
Quote
You write: “Software is shipped with untested parts every day, and just because it’s not entirely tested doesn’t mean it’s bad software ”
I hope you just got carried away and didn’t mean that. Or if you did mean that, then just because you can get away with shipping untested code doesn’t make it good code. Don’t you even aspire to some form of engineering discipline?
#10 by Daniel Ribeiro on August 21, 2010 - 7:28 pm
Quote
I like this kind of controversy. It is good to make people reaffirm their convictions, so that we do not keep repeating things like mantras, instead of actually thinking and considering practices for ourselves (Kent Beck shared a bit of his reframings on Startup’s Lesson Learned, which i commented on this other post: http://www.abetterstartup.com/2010/05/bootstrapping-versus-best-practice-in-lean-startups/). Sometimes I feel we don’t have near enough controversy.
On the other hand, it is interesting that this old thread not only caught up Reddit’s attention (as you mentioned on twitter), but also Michael Feather’s himself (http://michaelfeathers.typepad.com/michael_feathers_blog/2010/08/testng-and-what-wed-like-code-to-be.html, twitted by Kent Beck). He even responds with movies of both TestNG and Junit’s source code visualization.
Just hope you are not overwhelmed by the controversy, and keep the interesting posts coming.
#11 by SudarshanP on August 21, 2010 - 8:52 pm
Quote
How do you write a test for the implementation of e^x when there is no existing algo for the same. d(e^x)/dx = e^x. So take random x values and check whether exp(x+dx)-exp(x)/dx = exp(x).
To handle the finite precision of digital arithmatic, we can subtract LHS from RHS to see how big the difference is. The smaller the difference, the better the implementation. Before “your coder” gets to lay his hands on the code, a “mathematician” can “specify” what is acceptable and unaceptable from your function.
Where it is “possible” to be clear, it is better to “be clear” than “appear professional”. I wonder if a 10 page document that makes some vague description about the exponential function would be a tenth as useful as a clearly written test. That said, achieving clarity through a test is “pragmatism”. Thinking there is a silver bullet to all problems is stupidilty.
#12 by bestmore on August 21, 2010 - 9:24 pm
Quote
Are learning. It was accepted.
#13 by Joe on August 21, 2010 - 10:01 pm
Quote
If you look at unit tests like they are executable specs, yes, they may have bugs. You seem to state this as an argument for non-executable specs, word docs, activity diagrams, whatever. Wouldn’t you agree that non-executable specs are also prone to bugs, incompleteness and possibly even worse, staleness?
Personally, I try to write perfect code, but I don’t. You don’t. Nobody does. Leaving a trail of units tests at least lets me communicate to my fellow developers that, hey, my code isn’t perfect, but here’s my current understanding of how it should work. If you do something that breaks it, then we’re certainly not on the same page. Maybe you’re wrong, maybe I’m wrong, but there’s a conflict that needs to be resolved quickly before we start building on top of a potentially broken design.
#14 by Erik E on August 22, 2010 - 1:05 am
Quote
I diagree with the idea that so someone heretter propose that you don’t need comments. A lot of people who wrote the 3 mill loc app you are working on might have left the company long time ago. What are you supposed to do then? Reverse engineer the app every time you work on a new piece? Interfaces should be documented. At least describe what the purpose is if
I
I
#15 by Erik E on August 22, 2010 - 1:08 am
Quote
I diagree with the idea that so someone heretter propose that you don’t need comments. A lot of people who wrote the 3 mill loc app you are working on might have left the company long time ago. The purpose of classes should be documented. If that changes to frequently you must be doing something wrong. I like the philosophy of agle but agree it gets a bit dogmatic at times
#16 by Medwezys on August 22, 2010 - 1:32 am
Quote
Offtopic: the article is hardly readable on iPhone because of font-bg colors, you should consider changing some css if you want it to be usable for mobile users
#17 by Icelander on August 22, 2010 - 6:39 am
Quote
We’re all doing TDD every time we write code. Someone has expectations of what the program should be doing and we write code until it meets those expectations. In most places I’ve worked it’s been management having some vague idea of how it should work and programmers constantly asking them if they can stop working yet.
All unit testing does is provide a way for programmers to define the expectations before they start writing code, based on specs from management. It shortens the feedback loop so development can happen more quickly.
Now, if management changes their expectations or doesn’t allow enough time for writing test cases then of course TDD is going to look like a waste of time. But you’re going to have to check your code against something eventually, otherwise you’re writing random strings.
I don’t view tests as replacement for documentation, but as a way to make sure future modifications don’t break existing functionality. Recently I’ve been working on a very fragile code base with no tests. Most of my time was spent searching the code to find where to put my modifications so that I didn’t break the code. If there were
And the biggest improvement in my productivity has been my insistence on getting stakeholders to WRITE DOWN (along with a date) what they want done and how long I think it’s going to take. Most of the managers I’ve worked with have resisted this, and every project I haven’t done it for has either had huge scope creep or turned into a death march.
#18 by Mike on August 22, 2010 - 5:00 pm
Quote
A small nit:
“how do you specify an exponentiation function with a test?”
Easy: verify b^y = x log_{b} x = y
#19 by Mike on August 22, 2010 - 5:03 pm
Quote
Ugh, blog software stripped a rather important symbol out of that…
restated:
verify: b^y = x iff log_{b} x = y
#20 by J on August 22, 2010 - 7:38 pm
Quote
“Very often, Agilists simply forget that their job is to produce software that satisfies customers, not software that meets some golden software engineering scale.”
I am currently studying Agile Programming at university, and either Agile Programming has evolved in the last four years or you have missed a key element in Agile and Extreme Practices. When beginning the first iteration the FIRST pieces of implementation added are the ones that gives the customer the MOST value.
#21 by Pleb on August 22, 2010 - 7:43 pm
Quote
IMO, TDD is great for the first 5 minutes. Then it’s annoying. My version is TDWCWT. Think Design, Write Code, Write Tests. So, no the test don’t come last, but having thought about the design confirm while writing code that it works by writing the test.
To me the main advantage of unit testing is better code quality and the ability to rerun tests! Firing up the software to test a change is required, but counter productive if that’s the only method of testing.
#22 by Maxim Zaks on August 22, 2010 - 8:07 pm
Quote
Very interesting post and very interesting comments.
There are many good pro arguments for TDD and agile in the comments. But in my opinion the most important pro argument is Feedback.
XP is based on the idea of feedback loops. You get it from pair programming, testing, continuos integration a.s.o.
If you don’t test your code you will get feedback from your peer (if you making code reviews), QA (if you have one), or in the worst case from customer.
A test can be seen as customer that has some expectations for your product. And in the real world project there is always a customer first.
So that means:
test first = customer first
If you think about documentation. What is the easiest way to understand a product? Look how it is used. “Monkey see, monkey do” principal.
And as a test is a user, it describes you the usage.
#23 by Vital on August 22, 2010 - 8:29 pm
Quote
I have heard all sorts of excuses from people not wanting to write unit tests. And they did not write them at the end. With no exceptions, in 100% cases, the project ended up in if not being a total disaster, but very close to that. No matter how great the design was and how good the people were. This is the project itself, and I am not even mentioning the further supportability issues and technical depth created by famous “implement this feature quickly” approach. Also, in 100% cases my team, which is doing TDD, is way quicker in not just implementing those features, but actually getting them to work without breaking half of the other functionality.
Unit testing is used anywhere else in the industry. Only an idiot will install a hydraulic cylinder on the Airbus 380 without testing it first. Well, it can actually happen, but this idiot will be jobless in no time. Why the hell do software developers think they are special?
Software is one of the most complex engineering areas. As process, software development is prone to errors, and the biggest problem here is that the more time is passed since the mistake was made the harder it gets to locate it and fix. Unit tests eliminate about 95% of such errors; they also eliminate the need in debugging the application. I had several systems designed, implemented and commissioned without ever starting the debugger. It saved us an enormous amount of time.
Of course, on a downside, most people around us first found it hard to believe that software can just work at the first try. But they got used to it.
And, at the end of the day, if it’s worth writing, it’s worth testing. If it not worth testing, why bother then?
Also, Agile != TDD.
#24 by erazer on August 22, 2010 - 9:15 pm
Quote
This rings a familiar bell. Though I don’t agree with this blog entry entirely, there are similar situations that I come across frequently. I have seen developers not writing even a single line of a comment for an entire codebase (~20K). “Tests are the specs, the documentation” – that is a comment I often hear. I don’t agree that you have to spend minutes or hours understanding code where a few lines of comment that can be comprehended in a couple of minutes would suffice.
Vincent Massol’s work on Cargo (http://cargo.codehaus.org/) is one of the better disciplined works I have seen over years in open source space.
#25 by Davy Robertson on August 22, 2010 - 10:27 pm
Quote
Yes I think you are quite wrong here. You absolutley can test the whole solution. And why cant a test be a spec ? Most specs are usually wrong – if its in the form of a test – then its got clear defined inputs and outputs. You wouldnt get away without properly testing manufactured goods – difference is you test it during design and development. I think the problem is that software engineers like yourself are the ones who dont live in the real world. Still hanging on to the idea the software is like a movie – a creative product – thats perfect even with its imperfections – hogwash old boy – software quality is as important as the design. Both need to work together.
#26 by Petra Eckhart on August 22, 2010 - 10:47 pm
Quote
I totally agree with #23, saying: Agile !=TDD
Agile is a buzz word and while almost every company now claims to work agile, they all do it in a different way, which makes ‘agile’ a very fuzzy word, making it hard to claim if it is a good or bad way to program.
Do you want an agile development for your project? Ask if they can explain how they work, without using the word agile…. That gives you a hint weather it is good for your project or not. It depends on what they do and what kind of project you have.
TDD has lots of value… but please don‘t use it as a law which solves everything. PE, you shouldn’t forget the influence of the user (do usability tests) . If you got a client who is not that clear in describing what they want, I think making a rough untested version is a good thing: you can use it to find out if you are on the same track as the client. If you use a code-generation tool, writing tests for the CRUD actions is not a very useful action and if you write your own code, please do! Be pragmatic!
#27 by flukywotsit on August 22, 2010 - 11:25 pm
Quote
I have worked in both the old style development and with the more agile approach with scrum though i am yet to experience paired programming.
And i would never through away my technical requirements document as some agile approaches preach as its to useful for when office politics comes into play, I do develop with a sort of TDD approach, using scrum.
There will be developers out there that will want to hold on to the oldways and they do work, but i would say that agile though no were near perfect, it does kill off the need for a project manager and as some one who has worked in the public sector that is the best thing in the world.
Personally I think agile comes down to the quality of your team, i would not want to use it with a team of just out of uni developers, but adding one or two to a team of seasoned professionals. But a bunch of newbies working giving me a full design to look at, and unit testing there code as they go along with a little quality checks by a seasoned programmer could probably develop something good.
In the end bosses will aways want something yesterday, and be looking for what can be cut from the development time, so either way there will be times you will not be following any best practices process.
#28 by R00bN00b on August 22, 2010 - 11:49 pm
Quote
Amen to this article, for the most part. I do disagree about comments though, they’re usually crap and trying to figure out how the code following solves the outdated comment can make a 10 second glance at the code logic take 10 minutes. Perhaps some of us are more fluent in code than others? I can sure say I’d rather a chunk of java, C#, or php over a textual narrative any day. If you put VB in there then maybe I’d prefer the typo filled nonsensical narrative written in broken English that doesn’t appear to actually communicate anything at all.
I know the answer to the end-all-and-be-all of programming practices! Hire smart people, that are only territorially defensive when logic demands it. Get them awesome equipment and whatever tools they ask for, even if it’s old unsupported IDE software. Then get the frack out their way (and keep blonde bimbos with invented titles and physical education degrees out of their way too) and let them work wonders. You don’t even need a spec, smart people do cool stuff no matter what pile of turds of an idea they’re handed. A team of smart people do even cooler stuff. Add in one numbskull and that single vacuum of intellect will render all powers null and void. In other words fire more people you dodos. Hire smart people, fire dumb people. Easy peasy.
#29 by Richard Stanton on August 23, 2010 - 12:02 am
Quote
Agile (now known as “fragile” in our workplace) worked absolutely fine with project involving 2 programmers, 1 customer, a 7 page website. Even updating it 7 months later was no problem.
However, the methods failed miserably with 3 UI designers, 1 Systems analyst, 1 database architect, 1 spec author, 4 coders and 2.5 million lines of code. (actually, it failed way before we hit that point).
Horses for courses.
Great for low-risk startups and “programmed in a back-bedroom” smaller projects. And a much nicer way to code because things are developed more ‘on the fly’ (and as a coder I hate having to document things because I’m fundamentally lazy at heart!)
I believe Agile still beats pure waterfall methodologies even on larger projects; just inject some common sense and keep the “Objectives”, ERD diagrams and tester-checklist documentation in step with the code! (even better, get a spec-author to do the documentation for you
#30 by Nathan Manning - Nashville, TN on August 23, 2010 - 12:27 am
Quote
A while back I spent some time working at the NASA Data Center with a team that developed ESB integrations for the Constellation program within the Space Exploration Directorate. We had a quality problem, and decided to address it within the development team. I was charged with development white-box testing and regression testing. I ultimately turned that into a Continuous Integration Test & Build (CIBT) process. The team was supposedly already using TDD. I found that what unit test code that existed generally only tested what the developer thought would be valid input. There were no negative tests and no exception tests. Developers always do this – Assume their code will be receive only expected input.
The documentation and specs were even further out of date and that is just as bad. You cannot write a test plan without requirements documents and design specifications. And, Agile or not – these are still really a part of good engineering practice in any discipline. I have been saying for years – “Who would build any building without blueprints???”
Software projects would not exist with out a customer (internal or external) requirement or a market for a product. Simplified business processes that yield results or Satisfying Leisure computing are the only thing that produce our jobs.
Pragmatism and flexibility in achieving results while adhering to as much proven best practice as possible given the culture and management are what I have always practiced. Anything else belongs in an Ivory Tower.
#31 by Mike Gage on August 23, 2010 - 1:43 am
Quote
I couldn’t disagree more with this blog.
I have worked on many projects in many languages. The one I’m currently working on includes millions of lines of code. My experience is that the attitude that unit tests can be optional always leads to problems.
The people who have told me that agile doesn’t work have not really experienced agile, even though they may have thought that they had.
I can assure you that Agile works in the real world. It works very well.
#32 by Dan Kaminsky on August 23, 2010 - 1:54 am
Quote
OK, so I’ve spent some time working with a very large codebase.
The reality is that, once you enter the realm of thousands of developers, if you don’t have test cases you won’t have code that ships. This is because, at that scale, the game really shifts to discovering problems as soon as possible. Each layer always has dozens of other teams who, when they change things, could cause something to break. If that break isn’t discovered immediately by test cases, it’s found at the absolute end of cycle during final integration. By then, it can be way too late.
#33 by Steve on August 23, 2010 - 1:55 am
Quote
I’m afraid my testing practices would horrify a lot of people, but they work pretty well. I may specify test cases before coding, but generally not. My expected test results are almost always generated by running the code. After all, it’s often much easier to verify a result than to work it out for yourself (irrespective of P and NP). I find I get a lot more test cases much quicker that way, plus I almost always miss something working the results out manually, so I’d be throwing them away anyway when the code points out the mistake.
It’s a bit like those low level design documents, that almost always used to end up being written to match the code rather than visa versa – a case where I’ve long accepted Martin Fowlers view that the source is a design document (but not the only one).
Coverage stats tell me when I’ve (probably) got enough cases, though there are plenty of cases where 100% stats isn’t enough, and quite a few where 100% isn’t really achievable, and the verified test results get pasted into the regression test code so it stops complaining until something changes.
This is a price/performance kind of thing. I get a lot of value from writing tests – but you can’t ignore the fact it takes time to write the tests. Toy examples aren’t a problem so long as I can believe they scale reasonably and affordably. I won’t deny that very rarely, I’ve failed to spot an error in those “verified results”, but there’s no such thing as a proof that your code is perfect. Even formal proof tools rely on having some assumed-perfect alternative specification of what your code is meant to do, and frankly, I find it easier to get confused and make mistakes in math than I do in code.
There’s only one way I can really justify some of these extremist approaches. Given how many developers don’t really test at all, and how many workplaces will ban automated testing as timewasting then throw a tantrum when your slow tedious make-it-up-as-you-go-along manual testing misses an obvious regression, maybe the goal for some isn’t really to get people to do TDD or formal proofs. It’s a neat trick when you can get people to shift their behaviour in the general direction you want and believe not only that it was their idea all along, but that they proved you wrong by doing it. When you say “I got almost all the benefit for a tiny fraction of the cost”, someone may be struggling to hide their smug face.
#34 by Kevin on August 23, 2010 - 1:59 am
Quote
In my opinion TDD is used to drive design, not simply to get a list of green bubbles like some playground attraction as the OP appears to be implying, and contrary to Nathan Manning’s post, under-tested software can occur in any testing environment, it’s certainly not exclusive to TDD but it should be fairly evident from TDD as you’ll have a test/method ratio which approaches, or is less than, 1 as the project grows. Anyway the OP also seems to approve of releasing untested software:
“Software is shipped with untested parts every day, and just because it’s not entirely tested doesn’t mean it’s bad software or that the untested parts are “useless”.”
No, but I bet it’s a lot closer to bad software then the tested stuff and if it was code in the auto-pilot of the plane I was about to fly in… Well!
Agile methodolgies have been used in a number of successful, large and real-world projects, as I’m sure the OP could have ascertained for himself by simply casting around the internet. It sounds like the TDD presenter did both TDD and the attendees a disservice by presenting poorly; however, I’m more than a little suspicious of the OP, who claims to use TDD everyday but seems not to “get” the point of it at all.
All development practices come at a price as the OP rightly notes, the point is that many software projects hugely blow their budgetary or time constraints assuming they ever get released and then they are buggy and poorly designed. In my experience, working on large, real-world financial projects, Agile and TDD go a long, long way to relieving these problems.
#35 by Steve on August 23, 2010 - 2:34 am
Quote
@Kevin – the author *created* a testing framework. I don’t think his intent is to reject testing – only to point out a false absolute in someone elses argument. His rejecting someone elses claim does *not* automatically mean he’s taking the extreme opposite point of view.
#36 by Steve D on August 23, 2010 - 2:53 am
Quote
I think most of these comments sum up my thgouhts but I will add to them anyways. When anyone starts thinking that 1 size fits all without analyzing the situation you have done a disservice. you have moved away from “engineering”. I like the quote “When all you use is a hammer the whole world starts likking like nails.” Locking into 1 size fits all will never end up with the best end result. As an IT Director I see this in all areas of System and Development. Developers argue their language, framework, and methodology. System guys argue their OS. Whenever anyone argues their point withouht doing any type of investigation I immediately lose interest in that person’s service and discredit their abillity to server our company.
#37 by rei on August 23, 2010 - 3:37 am
Quote
TDD is a haphazard and slow way of figuring out what code has IO dependencies, what has side effects etc. and isolating them. The same thing can be achieved much more easily. Learn a pure functional language and apply that wisdom to your imperative code — and voila — highly refractorable code every time.
Most people do TDD without having any theoretical understanding of why the resulting code is consistently refractorable. They just do it because it tends to work out somehow. It’s a completely braindead, menial, insulting practice.
The “confidence” argument for TDD is absolutely weak for this reason. If you understood principles like function purity and side effects, you would be just as confident refractoring your code without TDD.
Also, claiming that TDD reduces bugs is only slightly better than saying that code that compiles will behave correctly. TDD IS NOT UNIT TESTING. It does *not* catch bugs, save for the absolute most superficial.
TDD is also a two-fold duplication of documentation: you end up with a test, its function name, and the proper English writeup. Sure, you’re not writing everything in the same way each time, but you’re an idiot if you don’t notice that on an abstract level you’re totally breaking DRY.
#38 by James Kosin on August 23, 2010 - 3:46 am
Quote
I think we are all missing the point. This is really the author’s exceptions to Agile.
I would have countered that Agile has the customer on site 24/7 to answer any small detail of the specification. Since the customer is the ultimate source of knowledge for the project.
James
#39 by Balaji B on August 23, 2010 - 3:59 am
Quote
Agile is good for therotical purposes only.
Till now I have not heard of a single project completed using Agile.
The so called non-technical english speakers have come up with a concept to keep themselves busy. If it was really that good, the industry would have replaced my favourite water fall model with this Agile.
But its not. Even after years of its inception.
What matters is how good software you develop within the deadlines. And not that you deliver a quality software 5 years late than expected with 5 times more cost and loss of business to. It would be useless then.
#40 by frank1914a4 on August 23, 2010 - 4:17 am
Quote
I agree with the author about viewing any methodology as a silver bullet, but some of his argument is just lame and boils down to, “but I don’t have time to do this so let me do whatever I have to in order to meet my deadline.”
I’ve got news for you, Agile or not, decades of that “I don’t care, just change it” mindset – far more than agile processes – are what’s responsible for the source code swamp many developers have to wade through daily.
Yes, a lot of Agile/XP practices assume you’re starting a small to mid-sized project from scratch and they don’t scale or graft onto larger legacy environments well. But to discard them as valuable unit test practices because they don’t meet your Big QA Lab definition of System Testing is being just as dogmatic as the evangelists.
Yes, software of any substantial size ships with bugs. This is where good managers have a role. They understand something many developers are often clueless about: that a dozen bugs in a lightly used functional area with adequate workarounds is less costly to a customer (and a vendor) than one bug in a highly used area with an unacceptable workaround.
Many low-priority bugs sit in the queue for years because they’re simply not worth fixing – they may be version or platform-specific or the customer base isn’t complaining, doesn’t justify it, or is due to be sunset.
Like it or not this is a reality of the non-Open Source software industry, no matter how hip or grey-suit that business may appear on the surface.
(Does anyone still wear grey suits?)
#41 by David Blake on August 23, 2010 - 4:33 am
Quote
Orthodox Agile may work, but I have seen few companies who actually follow all of the discipline required. Is is a downward spiral to cowboy coding. Even if a dev team did everything correctly, in this economy it’s not unusual for entire teams or divisions to be made redundant. How do you rely on pair programming when not a single person is left who worked on the code? Perhaps an out-sourced project is in-sourced, or a project is moved geographically from an office that is getting shuttered. Even an incorrect, out of date comment may be a godsend when you are taking on such a project and starting from zero.
#42 by Doug on August 23, 2010 - 5:12 am
Quote
A grossly myopic and egotistical article.
#43 by ncloud on August 23, 2010 - 5:58 am
Quote
I find good domain documentation to be indispensable because it describes what I need to know about the business. I find unit tests to be indispensable because they help me break dependencies and write terse code. I agree that tests are not a spec. BDD has always confused me because to me, unit testing is about the stability and design of a given slice of code, and not about business rules per se.
Unit testing can also be very useful in refactoring legacy code. We might have to get creative with interfaces and patterns to make newly tested code “fit” within the spaghetti, but it can work and it can buy you a lot in the long run. I remember gutting a payment processor implementation in a legacy app — I had to wrap the new code in “dummy” classes that still had the same interfaces as the old code, but inside the facade, my new implementation was loosely coupled and tested. My plan was to keep refactoring, little by little, pushing that facade further outward until I had every dependency broken and every class tested, but I got pulled into another project before it could be completed. But the payment processor worked, and never broke after that.
#44 by tim dugan on August 23, 2010 - 6:20 am
Quote
I think this is more of an endictment of agile evangelists then agile itself. “code that isn’t be tested is useless”? Of course not, but it is definitely riskier than code that is. Don’t take the hyperbole too seriously.
#45 by Jared on August 23, 2010 - 6:21 am
Quote
I love the rhetoric sentiment above. “If you haven’t drunk the Agile Kool-aid, you aren’t doing it right.” As if there is “a” way to do Agile. Wake up folks, do what works in your organization or work on figuring out what works and then do that. Agile is not a silver bullet, TDD is not always the best practice, working in absolutes will get you nowhere in software development. The world of software development is subjective, and correspondingly so is development methodology.
#46 by Samir Bellouti on August 23, 2010 - 6:35 am
Quote
I am agilist and I really believe that Agile is a very good methodology. However, Agile does not imply TDD neither does it imply pair programming.
Agile is about to comply and follow the values in the Agile Manifesto. Other than that are practices that claim to be implementations of Agile.
I am suspicious regarding TDD and Pair Programming. Even though I am a convinced Agilist (using SCRUM actually), after 14 years working in IT, I find it very hard to sell TDD to managers (cause it has an overhead whether or not some agilists will admit it) and worse, try to sell the idea of 2 programmers working on the same station…
To finish, one issue I do have at the moment with many presenters at such conferences is that it looks like they’ve lost contact with reality… Same with some authors. We need people on the field 365 days a year to talk about their experience not people reading books or just coming out from Universities labs with no clue about what’s going on in a real company.
My humble 5c point of view.
#47 by Paulo Morgado on August 23, 2010 - 6:37 am
Quote
And all examples are good only for user code.
None of the examples and reccommedations are for frameworks that are to be used by unknown developers and where the API is a deliverable.
#48 by James Walter Taylor on August 23, 2010 - 7:29 am
Quote
I think this diatribe has more to do with the wisdom in choosing this particular training, class or instructor. If the instructor can’t give you real examples which nail down the issues you’ve described they are not worth the time you spent.
The best of agile practices have real descriptions of how they work, how to introduce them to a team, and how some of these real issues of learning curve, new team members, introduction of new practices, u.s.w.; you’re describing an incompetent instructor.
#49 by Craig Berntson on August 23, 2010 - 7:53 am
Quote
I just read this for the first time. It’s right on the spot, but simply saying “Well, if you had started with TDD in the first place, you wouldn’t be having this problem today” is very naive.
Most of us work on existing code that others wrote sometimes years before, sometimes even before the concepts of Agile, TDD, and unit testing (as we know it today) existed. What we need is better information (articles, presentations, etc) on how to introduce unit testing to these legacy applications.
Jumping straight into TDD is a bit much for most organizations. I think a different approach of understanding the importance of unit testing and just how to create unit tests on existing code that’s being modified is the right approach.
#50 by mariangemarcano on August 23, 2010 - 8:08 am
Quote
About this “Software is shipped with untested parts every day, and just because it’s not entirely tested doesn’t mean it’s bad software or that the untested parts are “useless”.”
I think usually this untested software may not have unit tests or functional automated test but It is > 90 % likely that the developer has done a lot of manual / debugging and smoke testing and there is a team of manual testers behind on the component they are building, wouldn’t be nice if this test where reusable in the future?
I agree in the part that the Agilists should work more on present more realistic scenarios and how to deal with tone of code that doesn’t have reusable test, and the fact that the project’s needs varies a lot.