Sunday, July 22, 2012

The Law of Comparative Testing

Dear Blogspot,

Two hundred years ago, David Ricardo taught the world about the Law of Comparative Advantage.  It states that any two nations, groups, or persons who have differing relative productivity in the production of different goods always benefit from engaging in their most relatively productive activities, and then trading the produce thereof with each other.  It is a direct challenge to protectionism on a national scale, and self-sufficiency on a personal one.

This principle applies to the software world rather clearly when one considers the role of Developer and Tester (usually called Quality Assurance or "QA").  To explain both what the Law is, and how it applies to our subject, consider the following example:

Suppose a Developer is capable of producing 10 units of application code every day, or capable of reproducing 9 bugs per day.  Suppose further that a Tester is capable of producing 1 unit of application code every day, or capable of reproducing 5 bugs per day.  No matter who finds the bug, the Developer will have to produce more code to fix it, but we still generally consider Found bugs to be a good worth pursuing. Now consider all the possible scenarios:

  1. The Developer and the Tester only spend their time reproducing bugs: 
    1. 0 units of application code is written, and 14 bugs are found every day (but found in what?).
  2. The Developer and the Tester only write code: 
    1. 11 units of application code is written, and 0 bugs are reproduced (wheeee!).
  3. The Developer only reproduces bugs, the Tester only writes code: 
    1. 1 unit of code is written, 9 bugs are reproduced. (That is the safest program never released!)
  4. The Developer and the Tester write code half the time, and find bugs half the time: 
    1. 5.5 units of code written, 7 bugs found. (Serious context switching going on here)
  5. The Developer writes code full time, the Tester finds bugs full time: 
    1. 10 units of application code is written, 5 bugs are found and reproduced.
  6. The Developer writes code and tests half the time, while the Tester tests full time:
    1. 5 units of code written, 9.5 bugs found.

As you can see, the mixture of activities that produces the most code and the most found bugs is number 5, which should also be what any actual developer and tester should intuitively conclude.  Note that this is despite the fact that, in the example, the Developer is better at both writing code AND finding bugs! Many would dispute the productivity in finding bugs that I have assigned to my Developer, pointing out that "clear" or "white" box testing is generally inferior to black box, as knowledge of inner-workings can bias the production of test cases.  Fair enough, but that would only make the case more stark in favor of testing by Testers.


Another adjustment you might make to the example would be to calculate productivity not by units of time, but by dollar units in costs.  Since Software Development is, in the current market, a more scarce skill and in higher demand than Testing, it tends to cost more.  If the same test were run using dollar units, the results would lean even heavier towards the employment of Testers for testing in Software Engineering.

So, Blogspot, why would such an obvious argument even need to made?  Well, just as protectionism and self-sufficiency arguments crop up all the time in debates about international trade (witness the witless discussion of U.S. Olympic Team uniforms in 2012), they are starting to creep into discussions about methodology in Software Engineering as well. And for the same reasons, these discussions should be nipped in the bud, before it starts to mean less and more expensive software on the market for the rest of us.