[SystemSafety] Paper on Software Reliability and the Urn Model

[Peter Bernard Ladkin]

On 2015-02-25 11:37 , jean-louis Boulanger wrote:
> 
> For the software, no evaluation of reliability are acceptable or representative.

I think that is clearly wrong. I think my paper shows it is clearly wrong.

Software for protection systems in UK nuclear power plants has regularly used statistical
reliability assessment techniques for decades, and some of the best people in the field have worked
on them.

> Software contain bug (no idea of the number)
> the change process are not monitored (not the same team, not the same method, ...)

If the SW is being changed, then any statistical assessment has to start anew, for well-known
reasons. Statistical assessment is not at all practical for SW that is constantly changing.

> We don't have no operational history of software .... 

Well, you obviously can't do statistical assessment of reliability if there is no adequate
operational history. Actually, it's pretty hard to do *any* assessment of reliability if there is no
adequate operational history. And - please let's be clear about this - you really do need to perform
reliability assessment on critical software, be it statistical or whatever you can do.

>     Lots of things come up. People don't understand what the urn model has to do with software
>     evaluation. I have recently experienced reliability experts making incorrect claims, and non-experts
>     finding it difficult to adjudicate those claims.
> 
> 
> experienced reliability expert making incorrect claims because software reliability assessment is
> not a subject

I'll let you argue that with colleagues who have thirty or forty years doing it and getting their
work published in the premium forums on dependability in software.

> software are not reliable ... 

If that were generally so, then it shouldn't be running our planes, trains and cars, let alone our
safety-critical process plants. You've got a lot of campaigning ahead of you .........

But in fact lots of it is very reliable indeed.

The space shuttle control system software turned out to have been completely reliable.

In 27 years of fly-by-wire commercial aircraft, with thousands flying the skies every day, the
reliability of the control software, in the sense in which we mean it when using the Bernoulli
modelling, has been first-rate. There has only ever been one accident, when a few people were
injured. And that was due to a transient hardware fault, which was not filtered out by the control
SW even though the phenomenon was known, so that was a design decision, not a
Bernoulli-type-reliability issue.

The one incident which might have fallen into range was the 2005 Boeing 777 incident out of Perth,
which turns out to have been a configuration mistake. Such things wouldn't have been picked up by a
Bernoulli-type reliability assessment either.

The kinds of things which do keep some of us awake at night are generally not the kinds of phenomena
which could be identified through Bernoulli-type statistical assessment. They are the kinds of
things which fall through the gaps in specification.

> I am not an experienced mathematician but I understand that is not a good idea to apply the basic
> mathematics to a complexe product

On the contrary, if the product is complex, then basic math is what you want. Applying complex math,
there are so many ways to get the assumptions wrong without noticing.

>  For some software, I am the assessor from 10 years, and i confirm that the number of known bug
> increase after each version ....

Well, *we* occasionally do better than that :-)

PBL

Prof. Peter Bernard Ladkin, Faculty of Technology, University of Bielefeld, 33594 Bielefeld, Germany
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs.uni-bielefeld.de