[SystemSafety] Fwd: Comparing reliability predictions with reality

[Bev Littlewood]

[This is a resend of my posting from yesterday, which did not appear on the list because I sent it from the wrong email address 🙁. It was sent in response to Derek’s posting yesterday at 13.10. Derek did receive it, however, and his response also did appear on the list at 17.12. Apologies for the confusion! Bev]

> Begin forwarded message:
> 
> From: Bev Littlewood <bevlittlewood at icloud.com>
> Subject: [SystemSafety] Comparing reliability predictions with reality
> Date: 24 February 2025 at 16:08:49 GMT
> To: Derek M Jones <derek at knosof.co.uk>, systemsafety at lists.techfak.uni-bielefeld.de
> Cc: Bev Littlewood <bevlittlewood at icloud.com>
> 
> Hi Derek
> 
>> 
>> Having spent some time reading lots of papers on
>> models of software reliability, based on
>> cpu time between fault experiences, I have not found
>> one that measures the accuracy of the models.  By
>> accuracy I mean comparison of the prediction of time
>> to next fault against actual time to next fault.
>> 
>> There is something of a cottage industry of papers
>> that compare probability distributions fitted to historical
>> data.  Researchers seem shy about taking the next step of
>> comparing predictions of future faults.  Perhaps because
>> the results are not very good?
> 
> Perhaps you have not been looking in the right places, Derek! It’s about forty years since I published ways of doing these kinds of comparisons (i.e. of predictions against (later seen) observations). This work was reported in, e.g., IEEE Trans Software Engineering (hardly an obscure journal, I think you will agree), and various conferences, etc.
> 
> The easiest read for this stuff is in Michael Lyu’s old book (old, but still one of the best introductory accounts of this kind of material I believe):
> 
> ‘Techniques for prediction analysis and recalibration’ (with S Brocklehurst), Chapter 4 of The Handbook of Software Reliability Engineering (Ed. Michael Lyu), McGraw-Hill, New York, 1995, pp 119-166. (I think it is still available for free download)
> 
> More techie accounts are the original reports:
> 
> ‘Evaluation of competing software reliability predictions’ (with A A Abdel-Ghaly, P Y Chan), IEEE Trans. Software Engineering, SE-12, 9, September, 1986, pp. 950-967.
> 
> ‘Recalibrating software reliability models’, (with S Brocklehurst, P Y Chan, J Snell), IEEE Trans Software Engineering, Vol 16, No 4, pp 458-470, April 1990.
> 
> Very briefly, there are two main tools: u-plots, and prequential likelihood comparisons. The first of them is a kind of ‘absolute’ assessment of a single model's accuracy, based on a sequence of (prediction, outcome)-pair comparisons. It turns out that you can use this - somewhat surprisingly - to improve a model’s predictions, essentially by allowing it to learn from its past ‘errors’. The second is a means of comparing competing models to select the best (for a particular data set - i.e. sequence of inter-failure times). 
> 
> So solutions to the problem you pose have been around for years. I think my recalibration idea based on one of them is particularly neat - even if I say so myself….😀
> 
> I have not worked on reliability growth models for many years. It became a cottage industry producing new models with small tweaks. In fact you will still find papers on it in, e.g. IEEE Trans Reliability. But it long ago seemed to me that only very minor benefits in model accuracy were being obtained. And the techniques I have described above could often be used to check accuracy by applying several of the existing models.
> 
> I agree with you, though, that there continues to be a dearth of published data.
> 
> Cheers
> 
> Bev
> 
> _______________________________________________
> 
> Bev Littlewood
> Emeritus Professor of Software Engineering
> City St George's, University of London 
> 
> _______________________________________________
>                 
> 


_______________________________________________

Bev Littlewood
Emeritus Professor of Software Engineering
City St George's, University of London 

_______________________________________________
                

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20250225/d00d34b2/attachment.html>