This entry makes an interesting analysis of various ways that a build can
break.  As far as build philosophy is concerned, I have a very simple
motto:

Developers should never build.

Never.  Period.

Over the past ten years, I have worked at companies that manipulate huge code
bases on a daily basis, and their build system was so complex that teams of
several people to run it are pretty common.  All these companies are
experts at building software, but it’s amazing how so few of them really
understand how much time is wasted every time a developer needs to build the
entire product to get their job done.

Typically, developers will only be working on a very small fraction of the
code base, and they should only have to build this portion and nothing else. 
All the classes and external libraries that this code depends on to build
successfully should be downloadable in a binary form.

These "clean" snapshots should be generated by your continuous build system (CruiseControl
or similar) and can have several variations.  The two most important types
of snapshots in my opinion are:

  • Clean build.  The entire product built successfully but the
    tests have not been run, so some of them might fail.
     
  • Clean tests.  The entire product built successfully and
    passed all the required tests.

Typically, the label for a clean build will advance faster than that of a
clean test, therefore providing a more recent view of the product for those
developers that need the most up-to-date clean version of the build.  Also,
these two top categories can be declined in further subcategories (clean
check-in test, clean functional tests, clean partial build, etc…).

If you manage to set up such an infrastructure, a broken build becomes much
less harmful to the entire organization since there are very little instances
where a developer absolutely needs to synchronize to HEAD, which is the only
change list that can potentially be broken.  If developers only sync to a
clean label, they become completely shielded from occasional build breaks.

That being said, build breaks should be treated with the utmost emergency by
release engineers and I am more and more liking the idea that submissions that
break the build (and possibly, the tests) should be immediately and
automatically rolled back.  It might be a bit harsh, but it makes
developers more aware and more careful before submitting their code, because
undoing a rollback can sometimes be painful, depending on the source-control
system you are using (it’s trivial with Perforce, not necessarily so with
others).

Once such an infrastructure is in place, the daily routine of a developer
becomes:

  • About once a day, sync to a clean label and download the corresponding
    binaries.
  • Several times a day, sync only the subset of the project you are
    interested in if you need the latest bits (a step that’s most of the time
    even optional).

No more build break syndrome.