The Law of Comparative Testing

Dear Blogspot,

Two hundred years ago, David Ricardo taught the world about the Law of Comparative Advantage.  It states that any two nations, groups, or persons who have differing relative productivity in the production of different goods always benefit from engaging in their most relatively productive activities, and then trading the produce thereof with each other.  It is a direct challenge to protectionism on a national scale, and self-sufficiency on a personal one.

This principle applies to the software world rather clearly when one considers the role of Developer and Tester (usually called Quality Assurance or "QA").  To explain both what the Law is, and how it applies to our subject, consider the following example:

Suppose a Developer is capable of producing 10 units of application code every day, or capable of reproducing 9 bugs per day.  Suppose further that a Tester is capable of producing 1 unit of application code every day, or capable of reproducing 5 bugs per day.  No matter who finds the bug, the Developer will have to produce more code to fix it, but we still generally consider Found bugs to be a good worth pursuing. Now consider all the possible scenarios:

1. The Developer and the Tester only spend their time reproducing bugs: 
1. 0 units of application code is written, and 14 bugs are found every day (but found in what?).
2. The Developer and the Tester only write code: 
1. 11 units of application code is written, and 0 bugs are reproduced (wheeee!).
3. The Developer only reproduces bugs, the Tester only writes code: 
1. 1 unit of code is written, 9 bugs are reproduced. (That is the safest program never released!)
4. The Developer and the Tester write code half the time, and find bugs half the time: 
1. 5.5 units of code written, 7 bugs found. (Serious context switching going on here)
5. The Developer writes code full time, the Tester finds bugs full time: 
1. 10 units of application code is written, 5 bugs are found and reproduced.
6. The Developer writes code and tests half the time, while the Tester tests full time:
1. 5 units of code written, 9.5 bugs found.

As you can see, the mixture of activities that produces the most code and the most found bugs is number 5, which should also be what any actual developer and tester should intuitively conclude.  Note that this is despite the fact that, in the example, the Developer is better at both writing code AND finding bugs! Many would dispute the productivity in finding bugs that I have assigned to my Developer, pointing out that "clear" or "white" box testing is generally inferior to black box, as knowledge of inner-workings can bias the production of test cases.  Fair enough, but that would only make the case more stark in favor of testing by Testers.


Another adjustment you might make to the example would be to calculate productivity not by units of time, but by dollar units in costs.  Since Software Development is, in the current market, a more scarce skill and in higher demand than Testing, it tends to cost more.  If the same test were run using dollar units, the results would lean even heavier towards the employment of Testers for testing in Software Engineering.

So, Blogspot, why would such an obvious argument even need to made?  Well, just as protectionism and self-sufficiency arguments crop up all the time in debates about international trade (witness the witless discussion of U.S. Olympic Team uniforms in 2012), they are starting to creep into discussions about methodology in Software Engineering as well. And for the same reasons, these discussions should be nipped in the bud, before it starts to mean less and more expensive software on the market for the rest of us.

The Political Economy of Software

Dear Blogspot,

In the parlance of economics, "Rent" is defined as money one makes on some product or service above and beyond the cost of producing it. Importantly, it refers to money one makes on something that is not destroyed during consumption, and where true ownership does not change hands, so that, having rented it to one person, it can be rented again. It seems to me Blogspot, that software is clearly an area where the money is clearly made through rents.

In subscription based software services, this point is pretty clear. A consumer pays a periodic cost for access to the software, the value of which disappears after the service contract ends. Traditional software also fits this bill. Since the marginal production cost of producing the next unit of software is zero, the software per-se can be thought of as a single good. From this perspective, we see that once "sold" to one person, an identical unit can be "sold again". The important thing, however, is that, at some point, the money made on marginal units is over and above the production cost, which makes it "rent".

But how does this affect the political economy of software production houses? Suppose a software company endeavors to produce one product, without any updates (yes, no place would do that, due to the increased sales and rents that come with making the update, but stay with me anyway). During production of that software product, the productivity of the designers, developers, QA, etc. is vital. They and their knowledge constitute the true capital of the company, and embody its value in future sales. Once the product is released, however, this productivity and knowledge offer diminishing returns. In a perfectly fluid market where every employee makes their local marginal utility at all times, salaries of these workers would drop, and some, perhaps all, would be laid off. In fact, were it not for the fact that knowledge gained during development constitutes important capital going into an update release, we could imagine the entire development profession being made up of temporary contractors.

Given our example, however, after release, the software house almost immediately becomes a building full of rent-seekers. Developers would immediately change their goals from productive software outcomes to that of convincing the owner that he or she warrants a cut of the continued sales of the program they produced (for which time they've already been paid). This is pure politics and shenanigans.

Now, stepping outside the thought experiment back into the real world, Blogspot, my point becomes this: Even if software houses DO plan updates, or DO offer subscription services, to what degree do these phenomenon show up anyway? Just because the marginal value of continued development is not Zero does not mean that it approaches the value of the work done to produce the first release. And to that degree, one would expect more rent-seeking behavior out of developers than before. The difference in expected productivity might change the nature of the behavior from (in the Zero value case) one of complete fantasy to (in the Normal case) one of exaggerated claims. However, the behavior would still be there.

How do those whose salaries are already tied to current and future sales (investors, owners, perhaps upper management) respond to this realization? Do they realize it at all? That I often wonder.

PostScript:
Of all my letters to you Blogspot, this one seemed the most "out there" to me.  Well, maybe not .. See page 23 of the leaked Valve employee manual:

The relevant quote is on page 17:
"Valve is not averse to all organizational structure—it crops up in many forms all the time, temporarily. But
problems show up when hierarchy or codified divisions of labor either haven’t been created by the group’s members or when those structures persist for long periods of time. We believe those structures inevitably begin to serve their own needs rather than those of Valve’s customers. The hierarchy will begin to reinforce its own structure by hiring people who fit its shape, adding people to fill subordinate support roles. Its members are also incented to engage in rent-seeking behaviors that take advantage of the power structure rather than focusing on simply delivering value to customers."

Concentration Matters

Dear Blogspot,

I think I can speak for many programmers when I say that Concentration Matters... a lot. When coding, we often have to wrap our minds around some long train of data marshaling, subtle bits of transformation broken into discrete functions, and balancing the side effects of doing something in one process on some other process. This requires our brains to bring largish amounts of information into the fore of our mind and KEEP it there while we code, so that our inputs can be have a proper coarse plotted towards the desired output despite the complex set of shoals ahead. This act of keeping this broader information in mind, steering the line of code being written at the moment, while plotting the coarse immediately ahead is what I call "concentration". It requires a degree of patience and discipline. A flighty mind need not apply.

Imagine a widget machine that requires a long warmup, but slowly becomes more productive at generating widgets of higher quality over time as it reaches its peak rate in both speed and quality. Now imagine some idiot coming along and turning it off and on every 20 minutes. Welcome to the real world of software development.

For developers, these breaks of concentration take many forms: meetings (especially the impromptu sort), questions, favors, or even social visits. It is especially bad when the interruption comes from another developer who, in trying to maintain his own train of thought, interrupts another developer with a question. As an aside, I would not count developer initiated events such as web surfing, or writing blog entries in this list, since the concentration break has inner costs that sufficiently incentivizes against it. Moreover, it can be hard for non-developers to grasp this problem sufficiently to appreciate it. In my experience, few tasks in software engineering outside of writing code requires a level of concentration that makes it a serious matter to afford no mental breaks.

The inner costs I mentioned creates perverse incentives that adds to the general loss of productivity. After all, if you know you will have to bear the inner cost, and you know your productivity will be lower, why turn the "coding machine" back on at all? Better to wait until you have reasonable expectation of some period of quiet. For those of us who find the inner costs of the interruptions most jarring, better yet to work after everyone else has gone home. (Is this why so many programmers are also "night owls"? )

Occasionally, management will see the wisdom of this insight and take steps to mitigate the damage in order to increase productivity. Instituting "No meeting days", or "No interruption" periods during the day are interesting, if failed, attempts. Other members of the overall team, from QA to management all require information to effectively do their jobs; information that is inconveniently concentrated in the mind of the developer. In other words, attempts to fix this problem will invariably affect the productivity of others and the company overall. By how much? Who knows! Nobody bothers to measure.

So, Blogspot, once again I've proven a useless whiny writer, contemplating problems for which I have no proffered solution, except that these matters should be measured, and a degree of mitigation proper to its overall effectiveness instituted.

I would get right on that, but everyone is headed out to lunch, and I've work I can do now...

The Latest Development Fads...

Hey Blogspot,

Sorry I haven't written in awhile -- I'll have something new soon. In the meantime, I had this random observation today: given that time is money (especially in the software development business), would you Bank on anecdotal evidence?

When the next new technique rolls around, will the talk in the office be about the peer-reviewed studies showing its productivity benefits, or will it all start with "has anyone heard or tried out X"?

- Bo

The Marginal Unit of Testing

Dear Blogspot,

Here's a puzzle for you. If society benefits from the availability of water, why is a cup of water not provided everywhere? Sure, they will offer you one at eating establishments, but why not at bookstores, or electronics stores? Granted water is not free, but if it is so necessary for human survival, you'd think we would demand a glass upfront at every venue, just like we demand air to breathe wherever we go.

The reason we don't has to do with the costs versus the marginal benefit. Radio Shack would need to spend capital and recurring expenses to be able to offer something that most people would not even want to consume were it offered there. Meanwhile, the demand for drinks at restaurants is extremely high, due to the complementary good of Food that is consumed there. Put simply: the marginal benefit of that next glass of water, at most venues, is well below the cost of provision. Not at every venue, but at most.

This important economic insight applies equally to an important (and recently Hallowed and Revered, thanks to T.D.D.) object such as the Unit Test. A unit test, of course, is programming code written to test the functionality of other programming code. It is distinguished from integration or functional testing precisely because of the small segments of code that are the target of the unit tests. Unit Tests are almost always written by the same programmer who wrote (or will write) the code being tested.

Like water, unit tests are important. Also like water, the provision of unit tests is not free. However, unlike water, far too many developers (especially the last 5-10 years) would be Horrified to discover even one "venue" where unit tests are not provided for. This reflects, in my humble opinion dear Blogspot, a failure to think on the margin.

Like with Water and Radio Shack, spotting the most cost-inefficient places for unit tests is pretty easy: getters and setters, code that does trivial calculations, code that generates logs for developer purposes, or code whose failure is more cheaply spotted by integration testing (such as user interfaces), etc. Below this it gets rather fuzzy -- one might argue that code that has a high call count, has numerous dependencies, is modified frequently, has failed previously, or that performs a function whose failure would be disastrous are all great candidates for unit tests.

Either way, treating Unit Tests like Water and reasoning that "because they are important, they must be equally demanded in all situations" is both fallacious (see the Fallacy of Composition & Division) and a wasteful use of resources.

Price Shopping

Dear Blogspot,

Today I looked at web site statistics. I saw interesting data, including numbers of page views, typical page navigation flows, and similar metrics. The web developer then tied those metrics directly into how his automated web site testing tool was configured. The output from the testing tool, combined with the metrics, then directly informed the time he spent improving and optimizing specific aspects of the web site. This started me thinking about whether such statistics might be useful in directing all web developer tasks.

In markets, prices are signals of public preference. Higher prices signal higher preference, and often cause more production effort in that direction. For a free public web site, especially one dedicated to an unreleased product, there are no prices anywhere to be found! Therefore, like the apparatchik economists of the 20th century, web developers have only statistics to approximate this function.

So, does this work? Is this a good approximation? Not even close...

In the first place, while usage statistics reflect demand, the only cost incurred by the free web page viewer is their time. Suppose that an attempt was made to make the site profitable by charging the user a fixed price for each page click. The (few) remaining web users viewing habits would change considerably, and the usage statistics would be radically altered. Higher value pages would gain considerably relative to lower value pages. Users would likely use bookmarks to skip index/portal-like pages in favor of going directly to the pages they want most.

In the second place, while usage statistics aggregate access, they do not approximate the intensity of the desire for the various pages viewed. In economics, this is called price elasticity. This is why, in our hypothetical pay-per-click site, the fixed page price would cause those pages that people value below the fixed price, such as the index and portal-like pages, to lose traffic relative to other pages.

Lastly, and this goes back to a previous blog post, demand determined from site statistics can not be used to determine the profitability of spending more developer time on even the most-viewed pages. This is due to the difficulty in comparing the market-cost of a developer's time with the time spent on particular pages. Perhaps maintaining the most popular pages have developer time costs that make such work unwarranted. Perhaps the developer should spend his time instead on NEW pages. Without prices at both ends of the equation, it's impossible to say.

Web statistics are very useful things. For the purposes that this particular web developer was putting them to, they were very near perfect. However, if such direct demand signals are otherwise useless in directing web developer tasks, how much harder is it to determine the best use of the time of non-web developers, who don't even have that?

Wanted: VCC 9.5 For-Loop Expert, min 8yr

Dear Blogspot,

In the great span of events, it seems only yesterday that those who understood the mystical language of computers were called "Computer Programmers". Along with their fellow "Systems Administrators" and "Computer Operators", they formed the labor force behind the vital computing capital in large American corporations and governments.

My, but how times have changed.

Despite the fact that computers are in far more locations, Computer Operators have largely disappeared, due to the increasing ability of computer hardware to operate for indefinite periods without much attention. Systems Administrators have been broken into numerous specialized roles, dealing with upgrades, client software systems, security software, networking arrangements and components, monitoring, etc.

And likewise has gone the "Computer Programmer". Today, the market asks for specializations undreamed of in the past: skills with a specific programming language is minimal these days. Noone bothers to hire a C++ programmer for a PHP position. Knowledge of specific libraries, components, environments, and platforms also regularly appear on job descriptions. Even familiarity with specific development processes has entered demand.

So long ago, Adam Smith told us that "specialization is limited by the extent of the market".

The market for those who work on computers is large indeed.