[SystemSafety] The evidence base

[Olwen Morgan]

Little did I suspect that tossing in a featherweight contribution about 
cppcheck would lead to such serious discussions.

The problem of evidence for claimed good practice is important, so I'm 
here starting a new thread to focus on that issue.

In my experience, I've only once come across a software process 
sufficiently well controlled to give one confidence that 
measurement-based studies would produce repeatable, let alone 
reproducible, results. Most software processes I've seen are like the 
proverbial curate's egg - good in parts. IMO, therefore, it is nigh 
impossible to get hard evidence of the effectiveness of good practice 
unless the whole process is based on good practice. A clincher study 
would be if you were to examine a CMM top-level process before and after 
a good practice had been dropped. That way you could assess the effects 
of individual good practices separately, free from the noise that 
curate's-egg processes inevitably throw up. Naturally such an experiment 
would be costly and it is hard to see who would have the incentive to 
fund it.

I wondered if it might be possible to have something similar to a 
Cochrane collaboration for software process studies. For all I know 
there might be one? (Enlightenment please?). On the other hand, I 
strongly suspect that the difficulty of getting studies properly 
controlled in the first place would be a show-stopper.

IMO there is need for a debate to focus on the issue of quality of 
evidence and how we get good evidence. I have my differences with Derek 
J but on the question of evidence, I think he's right on metrological 
grounds (which may not be the grounds he thinks he has).

To my mind, the difficulty of setting up proper studies is similar to 
that faced by economists. They, like us, can only rarely do 
well-controlled experiments. Much of their work takes the form of 
post-facto analysis of data and is not controlled in the same way that 
experiments in the physical or life sciences can and usually have to be.

The question of whether using a certain practice saves money and effort 
is, I think, a different one but no less worth discussion for that. 
Room, perhaps for a code quality thread?


Olwen