Perfect Software: And Other Illusions about Testing

Gerald M. Weinberg (2008)
Review date: November, 2012
Opinion

I'm skipping the summary for this one; partly because I wrote this review in a hurry on an airport, but mostly because it would be too long. When I went through my notes and browsed through the book again, I realized how much it contains and how much I've already forgotten or failed to absorb.

To make a long story short, I really liked this book. A lot. I'm in a phase where I enjoy reading books written by industry veterans. They don't have to be very technical. I'm more curious about what they have done and seen, and what conclusions they have drawn based on that.

Jerry Weinberg is certainly an industry veteran. To make a point; on page 107-108 comes this quote: "That's why I'm a strong advocate of the test-first philosophy, whereby developers write their tests to include expected results before they write a line of code. It's what we did fifty years ago, but the practice was gradually lost when industry trends separated testing from development."

Now, of course you're interested in reading what a man who can make a statement like that has to say about software.

What struck me the most about this book is the way it talks about testing. My takeaway is that it describes testing as an activity the purpose of which is to obtain some information about the software. Whether the software functions as expected may be such information, but there's much other information to be discerned as well. The author phrases this as: "Is there at least one question about your product that testing can help answer?"

Later in the book, testing is broken down into more fine-grained activities like discovery, pinpointing, localization, determination of significance, repair. The purpose of this is to illustrate that testing isn't just a phase that comes towards the end of the project, close to a release. It's simply unfair to lump all of these activities as "testing" and say that "testing" delays the release. This break-down also helps in showing why testing becomes a challenge in bigger organizations, where these intertwined activities are split between different departments.

Exhaustive testing is another topic that pops up here and there. Early in the book it's stated that testing is sampling at best, and convincing arguments are given throughout the text. Now, personally I haven't been in a situation where somebody told me to do exhaustive testing, but after having read this book I'm even more convinced of how little we can actually test.

There's another recurring theme in this book as well - Meta testing; obtaining the information about the quality of information obtained by testing. The concept is introduced with examples of situations in which the organization and its communication don't seem to work well. These examples contain meta-information that may be helpful in diagnosing the testing performed in that organization. They may be statements similar to "it would take us a month to find our specs".

The book also devotes two chapters to a topic not very widely discussed, at least not in books - scams. Tool vendors may market their tools over-aggressively or promise too much, which produces one category of scams. The other category is maybe best described as "wishful thinking" or "brushing trouble under the carpet", in which case the scam is sort of self-created. What was interesting about these chapters is the fact that the author has experienced enough deliberate scams to actually elaborate about them in a book. That might serve as a warning.

Finally, the book also contains a people/psychology dimension. One chapter is pretty much about basal defense mechanisms like repression, rationalization, projection, and blame displacement. Similar in spirit is a chapter that contains various fallacies like: the blaming fallacy, exhaustive testing fallacy, testing produces quality fallacy, decomposition/composition fallacies, all-testing-is-testing fallacy, and finally the any-idiot-can-test fallacy. There's also a chapter on information intake and one chapter with more than 15 excuses for not doing reviews. For the purpose of discussion, I chose to group these under one umbrella. The presence of these chapters gives the book depth and they make it even more thought-worthy. Adding a people element spiced with some basic psychology to a book you'd place in the software shelf always makes you reflect on your experience of situations similar to what the author describes.

In summation, the book contains some not entirely shallow thoughts about what testing is and what it isn't, the notion of meta-testing, and a good dose of psychology. All in less than 200 pages. No wonder is a treasure chest.

All-in-all, I think this is a book of the same magnitude as "The Mythical Man-month". Both are small and easy to read, but there's no way to absorb everything in them in one go, so you have to reread them periodically. I really liked this book, and what I think made the biggest difference is the fact that it's short. Were it longer, it would just be too much.

I'm probably going to forget all details and insights, but this book has permanently expanded my view on testing.

Who should read this book

Those who want to widen their perspective on the meaning of testing are going to love this book.




News

  • 2015-09-29

    It's been almost one and a half year since I reviwed a book! I've been too absorbed by Writing my own. Anyway, I'm back with Jeff Patton's relatively...
  • 2014-01-04

    New category! Performance! Reviewed The Every Computer Performance Book. Check it out!
  • 2013-09-10

    Reviewed a book that' slightly less technical, but much more fun to read. It's I.T. Confidential.
  • 2013-08-13

    Reviewed yet another book on Visual Studio 2012 and TFS. I also created a "Microsoft" category and moved the other TFS book there from the "Tools"...
  • 2013-08-05

    Updated the FAQ. Included information about getting a book reviewed.