[SystemSafety] What do we know about software reliability?

[Peter Bishop]

I guess that is the nub of the argument with regard to systematic faults.

- as a standalone artefact, the software is "faulty" rather than reliable

As a component in some operating environment, E, "reliability" becomes
meaningful.

i.e. the probability of failure of the software component is

    Pr_E( not A)

i.e. *In environment E* , the probability of an input value that is (not A).

Now suppose we can specify which input values in a program are correct
or faulty (i.e. we know A and not A precisely).

could you tell me exactly when the program will fail in operation ?

- Obviously (apart from the special case where it always fails,  Pr_E(
not A)=1),
   you cannot do this unless you have a crystal ball that predicts
future inputs in environment E with total precision.

But you can express the future occurrence of A in probabilistic terms,
i.e. how often (not A) will arise in environment E.

In practice of course you do not actually have information about (not A)
- these are the bugs you have not found yet.

But testing in operational environment E allows you to set a confidence
bound on Pr_E(not A)
- even if you do not know where the bugs are located in the program
input space.

And that is what statistical testing does,
but as others have already said, the reliability bound is conditional on
the operating environment E.

Peter

PS

There has been some work on the worst case change in the bound that can
be claimed for a different operational profile E'
https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles

where idea of extra "padding tests" is introduced to maintain the
claimed bound for specified departures from E.

There is also an alternative strategy - "fair" testing - where testing
is based on the likelihood of a particular input A being faulty
(based on code path length rather than E), which can reduce sensitivity
of reliability bounds to profile change.

https://www.researchgate.net/publication/234802669_Rescaling_reliability_bounds_for_a_new_operational_profile

On 16/09/2020 01:02, hugues.bonnin at free.fr wrote:
> Hi all,
>
> I have an alternative "toy" to propose: do you think that this software is reliable (written in ada-like code)?
>
> begin
>
> if A then 
>   do_nothing 
> else
>   fail --potentially hurt and kill people
> end if
>
> end
>
> The specification of the software is to do nothing; 
> NB: I'm not asking if it is the best implementation, whatever the criteria are, but just : "is it reliable?"
>
> regards,
>
> Hugues
>
>
> ----- Mail original -----
>> De: "Peter Bernard Ladkin" <ladkin at causalis.com>
>> À: systemsafety at lists.techfak.uni-bielefeld.de
>> Envoyé: Mardi 15 Septembre 2020 19:58:45
>> Objet: Re: [SystemSafety] What do we know about software reliability?
>>
>> Bev and I and Dewi have a colleague who poses the following question.
>>
>> "We have clients who have installed hundreds of [examples of our kit]
>> over the last ten years, and
>> have never seen any failure. They want to use it in further systems
>> that they build. What arguments
>> do we/they need to provide in order validly to justify such further
>> use?"
>>
>> So, what is the answer to that question?
>>
>> PBL
>>
>> Prof. Peter Bernard Ladkin, Bielefeld, Germany
>> Styelfy Bleibgsnd
>> Tel+msg +49 (0)521 880 7319  www.rvs-bi.de
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>> Manage your subscription:
>> https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-- 

Peter Bishop
Chief Scientist
Adelard LLP
24 Waterside, 44-48 Wharf Road, London N1 7UX

Email: pgb at adelard.com
Tel:  +44-(0)20-7832 5850

Registered office: 5th Floor, Ashford Commercial Quarter, 1 Dover Place, Ashford, Kent TN23 1FB
Registered in England & Wales no. OC 304551. VAT no. 454 489808

This e-mail, and any attachments, is confidential and for the use of
the addressee only. If you are not the intended recipient, please
telephone 020 7832 5850. We do not accept legal responsibility for
this e-mail or any viruses.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20200916/a0151e51/attachment-0001.html>