Things like the Cone, the rising-cost-of-defects or the 10x claim have been kicking around for decades.
The evidence for or against TDD is, admittedly, inconclusive, but it's more recent and of a better academic caliber. There have been a lot of studies. Most of these studies aren't any good - but at least someone is trying.
There's a deeper question, which is "granted that all the empirical evidence we have so far for claims in software engineering isn't all that good, how can we get good empirical evidence?"
I suspect that the answer is going to involve changing the very questions we ask. "Does TDD work" is too fuzzy and ill-defined, and there's no way you can test that in a blinded random experiment. People's biases about TDD (subjects' or experimenters') are going to contaminate the evidence.
Instead, we need to ask questions that aren't susceptible to this kind of bias and contamination. For instance, we might want to unobtrusively study actual programmers working on actual projects, and record what causes them to write defects.
I fear the problems on this empirical approach are deeper.
The main problem is that software metrics are imprecise and non-objective. Lines of code, functional points, code coverage, counting code paths, ... we can't have a metric that can be accepted by everyone, all of them have big flaws. And the metrics are the basis for any reliable analysis, if we can't trust them we can't trust anything.
The second main problem is that it is very hard to isolate things under examination. How can we analyse TDD without taking into account the developer's grasp of good design (coupling and cohesion), Dependency Injection and Inversion of Control, refactoring techniques and tools, etc?
Software Engineering is a lot harder because it is much more akin to the fuzzy social studies (e.g.: economics, sociology, management) than to hard sciences (e.g.: computing science).
To be fair, the metrics you mentioned are objective. It's just debatable how relevant they are. Other metrics that are used are measures of coupling, cohesion, lines per method, methods per class, etc. There are probably more measures that are used. None of them are perfect, but they all help paint a picture that adds to our understanding.
To address your second point, it's actually not too difficult to isolate factors like TDD. The standard way is to have a control group and a test group. With a large enough of a sample size, you can determine statistical significance with standard tests.
Unfortunately, the test subjects are often university students, who are less experienced than professionals. The fact that data collected on students might not generalize to professionals is a threat to external validity, but should be made explicit in most papers. Most of the time, I think companies aren't very happy about having researchers use their engineers for experiments on the company's dime, but it does happen. So there are some papers out there reporting results with professionals.
> The evidence for or against TDD is, admittedly, inconclusive, but it's more recent and of a better academic caliber. There have been a lot of studies. Most of these studies aren't any good - but at least someone is trying.
Can you point me to some of those studies? Every time I look I only find the same 2 studies everyone quotes from (and aren't very good).
Things like the Cone, the rising-cost-of-defects or the 10x claim have been kicking around for decades.
The evidence for or against TDD is, admittedly, inconclusive, but it's more recent and of a better academic caliber. There have been a lot of studies. Most of these studies aren't any good - but at least someone is trying.
There's a deeper question, which is "granted that all the empirical evidence we have so far for claims in software engineering isn't all that good, how can we get good empirical evidence?"
I suspect that the answer is going to involve changing the very questions we ask. "Does TDD work" is too fuzzy and ill-defined, and there's no way you can test that in a blinded random experiment. People's biases about TDD (subjects' or experimenters') are going to contaminate the evidence.
Instead, we need to ask questions that aren't susceptible to this kind of bias and contamination. For instance, we might want to unobtrusively study actual programmers working on actual projects, and record what causes them to write defects.