[SystemSafety] RE : Qualifying SW as "proven in use"
C. Michael Holloway
c.m.holloway at nasa.gov
Thu Jun 27 15:13:40 CEST 2013
12.3.3 is about models, so it is referring to predictions of rates, not
actual rates.
On 6/27/13 8:59 AM, SPRIGGS, John J wrote:
>
> After saying in RTCA/DO-178C sub-section 12.3.3, quoted below, that
> "defection detection rate" is inadmissible, sub-section 12.3.4 allows use
> of "Actual error rates in the product service history", which I assume is
> the same as defect detection rate (assuming they fix the ones they find).
> Or does 12.3.3 mean prediction of the rate of detection of defects?
>
> John
>
> *From:*systemsafety-bounces at techfak.uni-bielefeld.de
> [mailto:systemsafety-bounces at techfak.uni-bielefeld.de] *On Behalf Of *C.
> Michael Holloway
> *Sent:* 27 June 2013 13:51
> *To:* systemsafety at techfak.uni-bielefeld.de
> *Subject:* Re: [SystemSafety] RE : Qualifying SW as "proven in use"
>
> The full DO-178C text in the section 12.3.3. 'Software Reliability Models'
> is as follows:
>
> Many methods for predicting software reliability based on developmental
> metrics have been published, for example, software structure, defection
> detection rate, etc. This document does not provide guidance for those
> types of methods, because at the time of writing, currently available
> methods did not provide results in which confidence can be placed.
>
> This text is a simplification of the text from DO-178B section 12.3.4
> 'Software Reliability Models':
>
> During the preparation of this document, methods for estimating the
> post-verification probabilities of software errors were examined. The goal
> as to develop numerical requirements for such probabilities for software in
> computer-based airborne systems or equipment. The conclusion reached,
> however, was the currently available methods do not provide results in
> which confidence can be places to the level required for this purpose.
> Hence, this document does not provide guidance for software error rates.
> If the applicant proposes to use software reliability models for
> certification credit, rationale for the model should be included in the
> Plan for Software Aspects of Certification, and agreed with by the
> certification authority.
>
> The removal in C of the "If the applicant ..." sentence could be
> interpreted as implying even less confidence today in the utility of
> software reliability models than was expressed in 1992.
>
> --
>
> C. Michael Holloway
>
> Disclaimer1: My opinions are mine alone. Give neither blame nor credit to
> my employer for them.
>
> Disclaimer2: The quotations from DO-178B/C are justified by the fair-use
> exception to copyright protection.
>
> On 6/27/13 7:40 AM, Nick Tudor wrote:
>
> There is adequate guidance within DO178C on the evidential requirements
> for previously developed software. My opinion is that it basically
> gives the hint that it is very difficult and that anyone considering
> doing it for anything above DAL C should re-consider. Also note that
> there is a statement in DO178C regarding software 'reliability' as
> follows: "Many methods for predicting software reliability based on
> developmental metrics (for example, software structure, defect
> detection rate, etc.) have been published. This document does not
> provide guidance for those types of methods, because at the time of
> writing, currently available methods did not provide results in which
> confidence can be placed."
>
> Cheers
>
> Nick Tudor
>
> On 27 June 2013 10:24, Matthew Squair <mattsquair at gmail.com
> <mailto:mattsquair at gmail.com>> wrote:
>
> Hi Bertrand,
>
> I think that you touch on one of the great problems of 61508 and it's
> children, that it becomes a distorting prism through which everyone
> views safety.
>
> There is always a lot one can do from a practical perspective but
> this is too often obscured by the obsfucation of ISO 61508
> artefact, the somewhat reflexive response of COMAH to the
> Buncefield accident being a case in point (http://wp.me/px0Kp-1BD).
>
> That being said, I do come back to the issue that Peter's original
> example has crystallised for me, which is that if you accept that
> random and systematic errors are different, then you should not
> expect the techniques to characterise one type of failure to
> automatically apply to the other.
>
> I'll tip my hat a little and add that I do think there's room for a
> "proven in use" argument as well...
>
> --
> Matthew Squair
>
> Email: MattSquair at gmail.com <mailto:MattSquair at gmail.com>
>
> Mob: +61 488770655 <tel:%2B61%20488770655>
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Thursday, 27 June 2013 at 6:07 PM, RICQUE Bertrand (SAGEM
> DEFENSE SECURITE) wrote:
>
> Let's put Peter's example in an more industrial and less
> academic frame.
>
> You are an instrumentist and automation technician. Your school
> background is bachelor + 2 years of professional training more
> or less on how to configure a PLC with graphical languages and
> to install pressure transmitters on pipes. You are working
> since ten years in a chemical plant. You know how to design a
> PID control loop for a level control and how to sequence bits
> of automatic control on your processes. Your boss recently sent
> you to a 3 to 5 days 61508 training session and now you are
> known in your company as a safety engineer. There is a plan for
> a revamping or an aextension of a part of your plant and your
> boss asks you to take care of all the safety systems because
> there are these new standards the plant has to comply with. All
> your usual suppliers come to propose you "certified safety
> equipments" but your boss refuse to add new items in the plant
> because of cost of adding new references in the stocks. So you
> are request to investigate how the existing equipment and
> architectures can be reused. Because this only a fictive
> choice, this has to be understood as: you are requested to find
> a way to demonstrate that the reused equipment complies to
> requirements equivalent to DAL B in avionics.
>
> May we add that you don't understand more than one word among
> five from what is debated in this forum, not even speaking
> about the concepts that go behind. In your mind, if an
> application involving a transmitter, a valve, a PLC housing a
> piece of software, that yourself or anybody else has coded five
> years ago, has never failed (failed not being clearly defined),
> there is no reason to believe that it might fail in the future
> if you apply it again in a rather similar way (similar not
> being better defined than failed).
>
> That's the issue, the context and the background for which we
> are now writing a standard.
>
> *Bertrand RICQUE*
>
> Program Manager, Optronics and Defense Division
>
> *T*+33 (0)1 58 11 96 82 <tel:%2B33%20%280%291%2058%2011%2096%2082>
>
> *M*+33 (0)6 87 47 84 64 <tel:%2B33%20%280%296%2087%2047%2084%2064>
>
> 23 avenue Carnot
>
> 91300 MASSY - FRANCE
>
> *http://www.sagem-ds.com <http://www.sagem-ds.com/>*
>
> **
>
> *Error! Filename not specified.*
>
> *From:*Matthew Squair [mailto:mattsquair at gmail.com]
> *Sent:* Thursday, June 27, 2013 9:19 AM
> *To:* RICQUE Bertrand (SAGEM DEFENSE SECURITE)
> *Cc:* systemsafety at techfak.uni-bielefeld.de
> <mailto:systemsafety at techfak.uni-bielefeld.de>
> *Subject:* Re: [SystemSafety] RE : Qualifying SW as "proven in use"
>
> I've been thinking about Peter's example a good deal, the
> developer seems to me to have made an implicit assumption that
> one can use a statistical argument based on sucessful hours run
> to justify the safety of the software.
>
> I don't think that's true, in fact I'd go further and say that
> whether you operate for a thousand hours or a million hours has
> no bearing on demonstrating software safety, because what we're
> interested in are systematic failures rather than random ones.
>
> Example, I have a piece of software and (despite my best
> efforts) there's a latent fatal fault within it, however
> testing hasn't discovered it and I'm also in luck in that the
> operating environment is sufficiently close to the test
> environment that the fault is not triggered in the operating
> environment. Now I could run the system for one, one hundred or
> a thousand years in that operating environment and I wouldn't
> see a problem. So according to the statistical treatment the
> software is safe, even with a fatal flaw isn't it?
>
> So logically if the number of hours you run in service in a
> particular environment has nothing to do with proving the
> safety of software, why couldn't I say that after one hundred
> hours the software was 'proven in use', for that specific
> environment. Why not one hour?
>
> In Peter's example the number of hours run on the original
> software version could have been one, or ten million and there
> still would have been the same end result, e.g a failure when
> put into a new operational context. In other words one hour of
> operations has as much weight as one thousand (in the same
> environment).
>
> We use hours as a measure of exposure when we test for random
> failures because they're well, random, but systematic failures
> aren't random so I don't think that using hours actually works.
> So what's the 'unit of exposure' that is valid for systematic
> failures?
>
> Another question, say I have developed a piece of software,
> it's now running in three quite different operating
> environments, in terms of evidence of 'safety' would I weight
> 300 hours of operation in a single environment the same as 100
> hours from each of these different environments? If so why?
>
> On Tue, Jun 18, 2013 at 6:41 PM, RICQUE Bertrand (SAGEM DEFENSE
> SECURITE) <bertrand.ricque at sagem.com
> <mailto:bertrand.ricque at sagem.com>> wrote:
>
> Hi Peter,
>
> Let's start some miles behind.
>
> There are 2 ways to claim that anything is compliant to 61508:
> 1 - Compliance. Yes, I know, it looks stupid as the way to
> claim compliance is nor ruled by the standard neither by the
> market. I can claim it on the ground of my reputation without
> having to bring any proof. Let's assume: on the basis of
> analysis made by firms having a strong reputation on the market.
> 2 - Proven use.
>
> Proven in use: what a nice idea.
>
> The standard is about safety. A manufacturer cand esign and put
> on the market a new equipment and get it "certified " for
> "compliance" for a given use (restricted operational and
> functional environment) for a "safety" purpose. Let's say
> safety critical pupose.
>
> Now let's go to your paper. First of all, you choose the
> example of a manufacturer, but we must rememeber that the
> standard adresses anybody. So a user could do it. Let's assume
> the user has more or less the same problems than the manufacturer.
>
> There are two possibilities for the previous use of the C
> equipment:
> 1 - previously used in a safety application with all the
> associated usual safety requirements (in particular
> dysfunctional requirements). Then it means obviously that the
> plant/application was NOT already compliant to 61508. Let's
> assume this is possible, although I doubt that plenty of users
> will shout it to the public...
> 2 - previously used in a non safety application. We can imagine
> that there are non-safety applications presenting very similar
> characteristics. I have however some doubts but lets assume it.
>
> >From a manufacturer point of view, it can be interesting to
> re-use C, most probably parts of C as C-SW, in new products.
> Honestly, I don't see any manufacturer going to the market
> saying: hey guys, here is my very old equipment that I have
> re-packaged and will sell you twice the price for safety
> applications with a nice SIL 3 stamp. The need is more like:
> hello, my dear certification company, here is my new product,
> it made of well known pieces, so please don't charge us too
> much for the certificate. You can consider C-SW as "proven in
> use", here are the data.
>
> >From an end-user point of view, (and this is what was in mind
> of the writers - "was" because it is becoming too demanding,
> and they are loosing the rules in 61511, pushing BTW 61508 for
> manufacturers only ...), if one can get rid of case 1, the
> issue is not to change anything in an existing plant and to,
> would an inspector ask strange questions, proove that the plant
> is the safest in the world and complies to any possible
> regulation. For competent and responsible end-users, the need
> it to have a framework to properly select equipments on sound
> basis and not on manufacturers data sheets.
>
> That's the context at business level. It is clear that points 1
> and 2 of Martyn, in a non regulated context, will remain a
> dream for ever...
>
> When it goes to technical level:
> * Concerning manufacturers, in the industries at the origin of
> the standard, I don't know a single one having a solid process
> to collect data from the users. Some adjust the calculated MTBF
> with the return rate for repair in the factories. Concerning
> electronic equipment, if an expensive piece of hardware is not
> attached to the electronic board, it is usually thrown to the
> garbage... Most of them don't knwo at alla where and how are
> installed the equipement. It is actually the opposite The users
> are the best organised to gather data (OREDA, EIREDA, etc...).
> So I think that the issue is very theoretical for manufacturers.
> * Concerning end-users, before even entering in your
> considerations (I fully support BTW), and remaining within the
> concerns of the writers of the standard (shaping a much
> narrower picture actully):
> - Most systems operate on demand and, due to the
> functional characteristics of the applications, experience very
> few demands (otherwise, it means that the plants are
> permanently in trouble. This means that the chances that there
> could be a reasonable quantitative basis to extrapolate
> anything seem to me very low.
> - What is happening now in IEC 61511 discussions (with
> more or less the same persons as the ones at the origin of IEC
> 61508) is that three parties are building themselves:
> 1 - the ones (mostly end users) thinking that the concept
> dosn't apply to "re-use" but only to "already installed". They
> argue that if it is working properly, it will continue so. They
> keep a low profile on "re-use" but I suspect them to focus the
> discussion on "already installed" and use the concept for
> "re-use" later as it would not be clearly excluded in the
> standard ... They are also the ones that, rightly, don't want
> to buy expensive "certified" equipment with a lower performance
> than equipment they already have in their plant. The issue is
> real and interesting. These ones are mainly eurpoeans.
> 2 - the ones (mostly end-users) who want to re-use equipment
> because they want to reduce the number of different equipment
> in a plant (they are also strong opponents to diversity because
> of costs). These ones are mainly North American.
> 3 - the manufacturers are mainly against because they want to
> sell new equipment and they don't see how to actually "prove in
> use" as they lack data. The consultants are against because
> they don't where is the rationale (exactly what your paper is
> about).
>
> I am rather pessimistic about the outcomes in 61511. As far as
> 61508 is concerned, my opinion, converging more and more with
> several opinions expressed here is that:
> * Concerning software, the requirements must obviously be very
> stringent
> * This will implicitely so much limit the applicability of the
> concept that it will become useless for end-users and PLCs,
> * May-be it will be applicable for very tiny parts of firmware
> for manufacturers
> * It will de-facto create an insconsistency with 61511 if 61511
> doesn't align on 61508 requirements
>
> Bertrand Ricque
> ________________________________________
> De : systemsafety-bounces at techfak.uni-bielefeld.de
> <mailto:systemsafety-bounces at techfak.uni-bielefeld.de>
> [systemsafety-bounces at techfak.uni-bielefeld.de
> <mailto:systemsafety-bounces at techfak.uni-bielefeld.de>] de la
> part de Peter Bernard Ladkin [ladkin at rvs.uni-bielefeld.de
> <mailto:ladkin at rvs.uni-bielefeld.de>]
> Date d'envoi : lundi 17 juin 2013 12:32
> À : systemsafety at techfak.uni-bielefeld.de
> <mailto:systemsafety at techfak.uni-bielefeld.de>
> Objet : [SystemSafety] Qualifying SW as "proven in use"
>
>
> Folks,
>
> there is a significant question how SW can be qualified as
> "proven in use" according to IEC
> 61508:2010. There is a judgement in some quarters (notably the
> German national committee) that the
> criteria in IEC 61508:2010 are inappropriate. I think it
> wouldn't be out of place to say that many
> in the IEC 61508 Maintenance Teams find the current criteria
> unsatisfactory in one way or another.
>
> We in Germany have been discussing the issue and possible
> solutions for a couple of years, and
> recently the discussion has gone international. There seems to
> be a general feeling that qualifying
> SW statistically via the approach given by the exponential
> failure model is not practical, because
> the data requirements are overwhelming - it is regarded by most
> as implausible that companies will
> have the requisite data to the requisite quality even for SIL
> 2. But even if you qualify your SW for
> SIL 2 or higher without such data, then at some point some data
> will exist and people use such data
> as evidence that the original assessment was accurate. But what
> sort of evidence does it offer? The
> answer is probably a lot less than you might be convinced it does.
>
> There seems to me to be a lack of examples where things can go
> wrong - at least a lack of examples
> specifically adapted to assessments according to IEC
> 61508:2010. So I wrote one up - fictitious but
> I hope still persuasive - to illustrate what (some of) the
> assurance issues are. I hope it can aid
> the debate.
>
> http://www.rvs.uni-bielefeld.de/publications/WhitePapers/LadkinPiUessay20130614.pdf
>
> PBL
>
> --
> Prof. Peter Bernard Ladkin, Faculty of Technology, University
> of Bielefeld, 33594 Bielefeld, Germany
> Tel+msg +49 (0)521 880 7319
> <tel:%2B49%20%280%29521%20880%207319> www.rvs.uni-bielefeld.de
> <http://www.rvs.uni-bielefeld.de>
>
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent
> contenir des informations confidentielles ou ayant un caractère
> privé. S'ils ne vous sont pas destinés, nous vous signalons
> qu'il est strictement interdit de les divulguer, de les
> reproduire ou d'en utiliser de quelque manière que ce soit le
> contenu. Si ce message vous a été transmis par erreur, merci
> d'en informer l'expéditeur et de supprimer immédiatement de
> votre système informatique ce courriel ainsi que tous les
> documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain
> confidential or proprietary information. If you are not the
> intended recipient, you are notified that any dissemination,
> copying of this e-mail and any attachments thereto or use of
> their contents by any means whatsoever is strictly prohibited.
> If you have received this e-mail in error, please advise the
> sender immediately and delete this e-mail and all attached
> documents from your computer system."
>
> #
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>
>
>
> --
> *Matthew Squair*
>
> Mob: +61 488770655 <tel:%2B61%20488770655>
>
> Email: MattSquair at gmail.com <mailto:MattSquair at gmail.com>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent
> contenir des informations confidentielles ou ayant un caractère
> privé. S'ils ne vous sont pas destinés, nous vous signalons
> qu'il est strictement interdit de les divulguer, de les
> reproduire ou d'en utiliser de quelque manière que ce soit le
> contenu. Si ce message vous a été transmis par erreur, merci
> d'en informer l'expéditeur et de supprimer immédiatement de
> votre système informatique ce courriel ainsi que tous les
> documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain
> confidential or proprietary information. If you are not the
> intended recipient, you are notified that any dissemination,
> copying of this e-mail and any attachments thereto or use of
> their contents by any means whatsoever is strictly prohibited.
> If you have received this e-mail in error, please advise the
> sender immediately and delete this e-mail and all attached
> documents from your computer system."
> #
>
> _______________________________________________
>
> The System Safety Mailing List
>
> systemsafety at TechFak.Uni-Bielefeld.DE
> <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>
>
>
> --
> Nick Tudor
>
> Tudor Associates Ltd
>
> Mobile: +44(0)7412 074654
>
> www.tudorassoc.com <http://www.tudorassoc.com>
>
> Description: Image removed by sender.
>
> *77 Barnards Green Road*
>
> *Malvern*
>
> *Worcestershire*
>
> *WR14 3LR
> *Company No. 07642673**
>
> *VAT No:116495996*
>
> *www.aeronautique-associates.com <http://www.aeronautique-associates.com>*
>
>
>
> -----------------------------------------------------------------------------
> If you are not the intended recipient, please notify our Help Desk at Email
> Information.Solutions at nats.co.uk immediately. You should not copy or use
> this email or attachment(s) for any purpose nor disclose their contents to
> any other person.
>
> NATS computer systems may be monitored and communications carried on them
> recorded, to secure the effective operation of the system.
>
> Please note that neither NATS nor the sender accepts any responsibility for
> viruses or any losses caused as a result of viruses and it is your
> responsibility to scan or otherwise check this email and any attachments.
>
> NATS means NATS (En Route) plc (company number: 4129273), NATS (Services)
> Ltd (company number 4129270), NATSNAV Ltd (company number: 4164590) or NATS
> Ltd (company number 3155567) or NATS Holdings Ltd (company number 4138218).
> All companies are registered in England and their registered office is at
> 4000 Parkway, Whiteley, Fareham, Hampshire, PO15 7FL.
> -----------------------------------------------------------------------------
--
C. Michael Holloway, Senior Research Engineer
Safety Critical Avionics Systems Branch, Research Directorate
NASA Langley Research Center / MS 130 Hampton VA 23681-2199 USA
office phone: +1.757.864.1701 /forwarded to/ google voice: +1.757.598.1707
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20130627/f8539bf0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 425 bytes
Desc: not available
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20130627/f8539bf0/attachment-0001.jpe>
More information about the systemsafety
mailing list