[SystemSafety] What do we know about software reliability?

Tue Sep 15 10:45:55 CEST 2020

On 14/09/2020 15:04, Martyn Thomas wrote:
>
> Why are you completely dismissing software reliablity?
>
> Is it not the case that if you can tolerate a failure rate of once in
> 1000 hours, 99% confidence through testing would take about 200 days
> to demonstrate (so long as the test environment is "sufficiently" like
> the future operating environment and you are able to detaect every
> failure correctly)?
>
And statistical testing is used in the UK nuclear industry fore safety
critical systems, so it is not just abstract theory,

Re your characterisation of confidence based statistical testing on P153
(with no reference), I do not think it is fair to dismiss this because
"p can vary by orders of magnitude". Testing presumes a fixed
operational profile and a constant probability of failure.

There has also been some work on the impact of profile change on the
bound that can be claimed.

https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles

BTW, re, your summary of my paper on the same page, I think you missed
the main point. This is a*predictive* theory to derive a worst case
bound for some time in the future, i.e.

Given N faults what is the worst possible reliability  at some future
time T?
- it assumes fault fixing  will occur during that time.

You also only presented the theory of N=1, and you seem to assume the T
has already happened with zero failures (not a requirement for this model)

Might have been better to reference the original worst case bound
version (which makes it clear that it is a long term forward prediction)

https://www.researchgate.net/publication/3152200_A_conservative_theory_for_long-term_reliability-growth_prediction

> Of course, the testing would have to be repeated following a change to
> the software, unless you have enough formality to show that the change
> cannot affect reliability.
>
> In specific circumstances, you can do better than this. Bev
> Littlewood's published papers provide strong evidence and a rich
> bibliography. Bev's paper on "How reliable is a program that has never
> failed?" offers a useful rule-of-thumb: that aften n hours of fault
> free operation, there is about 50% chance of a failure in the
> following n hours (subject to some obvious constraints).
>
> The difficulties rapidly escalate when you need 10^-4 or better at
> >90% confidence.
>
> Martyn
>
> On 14/09/2020 14:14, SPRIGGS, John J wrote:
>>
>> In my experience, if Software Reliability is mentioned at a
>> conference, at least one member of the audience will laugh, and if it
>> is mentioned in a work discussion, at least one member of the group
>> will get angry.
>>
>> Interestingly, some of the same people who say it is impossible to
>> quantify software failure rates will set numerical requirements for
>> Software Availability – if you get one of those, ask the Customer how
>> (s)he wants you to demonstrate satisfaction of the requirement.
>>
>>  
>>
>> John
>>
>> *From:*systemsafety
>> <systemsafety-bounces at lists.techfak.uni-bielefeld.de> *On Behalf Of
>> *Derek M Jones
>> *Sent:* 14 September 2020 12:54
>> *To:* systemsafety at lists.techfak.uni-bielefeld.de
>> *Subject:* [SystemSafety] What do we know about software reliability?
>>
>>  
>>
>> All,
>>
>> What do we know about software reliability?
>>
>> The answer appears to be, not a lot:
>> http://shape-of-code.coding-guidelines.com/2020/09/13/learning-useful-stuff-from-the-reliability-chapter-of-my-book/
>> <http://shape-of-code.coding-guidelines.com/2020/09/13/learning-useful-stuff-from-the-reliability-chapter-of-my-book>
>>
>> -- 
>> Derek M. Jones Evidence-based software engineering
>> tel: +44 (0)1252 520667 blog:shape-of-code.coding-guidelines.com
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>> <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>> Manage your subscription:
>> https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
>>
>>
>>
>> ------------------------------------------------------------------------
>> If you are not the intended recipient, please notify our Help Desk at
>> Email Information.Solutions at nats.co.uk immediately. You should not
>> copy or use this email or attachment(s) for any purpose nor disclose
>> their contents to any other person.
>>
>> NATS computer systems may be monitored and communications carried on
>> them recorded, to secure the effective operation of the system.
>>
>> Please note that neither NATS nor the sender accepts any
>> responsibility for viruses or any losses caused as a result of
>> viruses and it is your responsibility to scan or otherwise check this
>> email and any attachments.
>>
>> NATS means NATS (En Route) plc (company number: 4129273), NATS
>> (Services) Ltd (company number 4129270), NATSNAV Ltd (company number:
>> 4164590) or NATS Ltd (company number 3155567) or NATS Holdings Ltd
>> (company number 4138218). All companies are registered in England and
>> their registered office is at 4000 Parkway, Whiteley, Fareham,
>> Hampshire, PO15 7FL.
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-- 

Peter Bishop
Chief Scientist
Adelard LLP
24 Waterside, 44-48 Wharf Road, London N1 7UX

Email: pgb at adelard.com
Tel:  +44-(0)20-7832 5850

Registered office: 5th Floor, Ashford Commercial Quarter, 1 Dover Place, Ashford, Kent TN23 1FB
Registered in England & Wales no. OC 304551. VAT no. 454 489808

This e-mail, and any attachments, is confidential and for the use of
the addressee only. If you are not the intended recipient, please
telephone 020 7832 5850. We do not accept legal responsibility for
this e-mail or any viruses.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20200915/116b0b85/attachment.html>