[SystemSafety] Fwd: NYTimes: The Next Accident Awaits

Tue Feb 4 11:25:51 CET 2014

I agreed, but goals can be specified in different ways (here are two common
ones):
     1. "The system will have a 10^-x probability of reaching a hazard
state over [some performance period]."
     2. The developers will identify the hazards and their causes and
eliminate and mitigate them.

These different goals lead to different ways to try to reach them and
convince ourselves that we have done so well enough to allow use of the
system. The term "goal oriented" is primarily being used for goals of type
1 above.

By the way, "prescribing methods" is not necessarily the only way to
achieve goals. Again, one of the reasons I like MIL-STD-882 is that it does
not prescribe specific methods. It simply describes processes that must be
followed. For example, one task is a hazard analysis, with the types of
results such a hazard analysis must produce delineated. Specific methods of
doing the hazard analysis (such as fault trees or HAZOP) are not specified.
That way, progress is not delayed by requiring people to use obsolete
methods as in some other standards until years (sometimes decades) are
spent in updating standards, which by that time are already obsolete again.

>> This is, BTW a debate, at the core of IEC 61508...
Absolutely.

Nancy

On Tue, Feb 4, 2014 at 5:07 AM, RICQUE Bertrand (SAGEM DEFENSE SECURITE) <
bertrand.ricque at sagem.com> wrote:

> Nancy I tend to agree with you about the necessity of sound methods to
> move further towards a goal based approach.
>
>
>
> And then you suggest prescription.
>
>
>
> And we are back to the ambiguity on semantics already discussed with
> "safety case".
>
>
>
> When you write prescription, you seem to think, tasks and methods. There
> are industries that would understand "use of limited lists of components
> avoiding spilling engineering hours on safety" as it was in EN954 for
> instance !
>
>
>
> There are other that would understand "if you prescribe methods, it is to
> reach goals, and therefore it is goal oriented".
>
>
>
> And finally you have others understanding "a safety case (as a document)
> is the proof that reach goals. The methods don't prove anything."
>
>
>
> This is, BTW a debate, at the core of IEC 61508...
>
>
>
> Bertrand Ricque
>
> Program Manager
>
> Optronics and Defence Division
>
> Sights Program
>
> Mob : +33 6 87 47 84 64
>
> Tel : +33 1 59 11 96 82
>
> Bertrand.ricque at sagem.com
>
>
>
>
>
>
>
> *From:* systemsafety-bounces at lists.techfak.uni-bielefeld.de [mailto:
> systemsafety-bounces at lists.techfak.uni-bielefeld.de] *On Behalf Of *Nancy
> Leveson
> *Sent:* Tuesday, February 04, 2014 11:00 AM
> *To:* Andrew Rae
> *Cc:* <systemsafety at techfak.uni-bielefeld.de>
> *Subject:* Re: [SystemSafety] Fwd: NYTimes: The Next Accident Awaits
>
>
>
> Andrew Rae writes:
>
>
>
> >>4) Safety cases aren't a substitute for competent review, and in fact
> increase the need for strong adversarial review
>
> >>    [I think we have a range of non-contradictory opinions here. E.g. I
> maintain that all safety assurance needs strong review, I think Nancy is
> holding that strong
>
> >>   prescription reduces the effort required for this review.]
>
>
>
> Effort is not my major concern. I am concerned that we are moving toward
> goal-based approaches without having the engineering techniques to
> determine whether the goals have been achieved. Review will not solve the
> problem when our analysis methods are inadequate. And my concern about
> confirmation bias arises here in the review process.
>
>
>
> I just finished a research project involving a new standard for air
> traffic control. The standard was jointly created by agencies and experts
> on both sides of the Atlantic. The list of people who produced and vetted
> it is very long. It is a goal-based standard. The analysis that supported
> this standard (included in the appendix) was dangerously incomplete (we
> easily found important missing cases) [when I first saw it I was appalled]
> and probabilities were pulled from thin air, particularly about human
> behavior. It was based on fault trees and event trees and the classic
> assumption that accidents are caused only by component failures. They
> didn't even identify those correctly.
>
>
>
> People can differ on what they believe the solution is here. My preferred
> solution, until we have a better one, is to use an approach like the one in
> MIL-STD-882 which requires various tasks, including hazard analysis, as
> well as a plan for applying the tasks. Is this approach perfect? No. But
> switching to a goal-based approach before we have the engineering methods
> to determine whether the design will achieve those goals, is worse in my
> opinion.
>
>
>
> Nancy
>
>
>
> On Tue, Feb 4, 2014 at 4:37 AM, Andrew Rae <andrew.rae at york.ac.uk> wrote:
>
> I think it is pretty clear (as Peter has illustrated well through his set
> of examples) that we all may be using "safety case" as shorthand
> for a number of different things relating to design, regulation,
> operations and assurance.
>
> I submit that there are a number of concerns common to all of these, which
> may be minimised or exaggerated depending on exactly what is meant.
>
> 1) A purely goal-based approach may result in failure to follow good
> design principles and/or safety processes.
>
>     [The extent to which this is true depends on whatever prescription
> goes along with the safety case, and the ability of the regulator to spot
> deviations from good practice]
>
> 2) A purely goal-based approach can shift the focus of safety efforts to
> quantitative assessment of safety instead of good design. This is a Bad
> Thing.
>
>     [The extent to which this is true depends on the industry approach to
> goal-setting and demonstration that goals are met]
>
>
>
> 3) An explicit safety argument may increase confirmation bias, making good
> review harder
>
>     [If true, this would apply to all the meanings of safety case, but
> this is the one we have a scientific disagreement about]
>
> 4) Safety cases aren't a substitute for competent review, and in fact
> increase the need for strong adversarial review
>
>     [I think we have a range of non-contradictory opinions here. E.g. I
> maintain that all safety assurance needs strong review, I think Nancy is
> holding that strong prescription reduces the effort required for this
> review.]
>
>
>
> 5) An explicit argument may help in cases where there are degrees of
> freedom in applying regulation and standards, at least to the extent that
> it requires people to make choices sensibly and justify them
>
>     [This would apply when safety cases are attached to prescribed safety
> processes, would not apply for purely goal based situations, and would be
> irrelevant where design was prescribed]
>
>
>
> 6) Customary practice and culture may in fact swamp the effects of choices
> about how to regulate or present assurance evidence. Safety cases aren't a
> direct solution to entrenched cultural problems.
>
>    [Caveat is that they are often introduced as part of sweeping reforms -
> hard to spot the signal from the noise here]
>
> Have I misrepresented anyone here or missed any major concerns?
>
> Drew
>
>
>
>
>
>
> My system safety podcast: http://disastercast.co.uk
> My phone number: +44 (0) 7783 446 814
> University of York disclaimer:
> http://www.york.ac.uk/docs/disclaimer/email.htm
>
>
>
> On 4 February 2014 09:12, Peter Bernard Ladkin <
> ladkin at rvs.uni-bielefeld.de> wrote:
>
>
>
> On 2/4/14 1:42 AM, Nancy Leveson wrote:
> > were simply the first 15 references that came up when I googled "safety
> case" and "safety case
> > regime." .....It would be best to
> > create a new term rather than use a term that has been defined for
> decades and is widely used to
> > have a specific meaning..
>
> Oh, my!
>
> A random Google search for the term "safety case" is as least as likely to
> turn up this discussion
> as anything else. That's not necessarily how you go about finding out what
> the term might mean. (My
> search turns up Nancy's paper in the first fifteen. Given the current
> discussion, I am tempted to
> regard that as *proof* that such a search isn't going to tell you what the
> term might mean :-) ).
>
> A *goal-oriented* (rather than random Google) search in the WWW for the
> notion of "safety case"
> could profitably start with the UK HSE, since Lord Cullen is the source
> both for the original
> recommendation concerning the notion of "Safety Case", in those words, and
> the origin of the HSE.
>
> One finds, on searching the HSE WWW site for the term "safety case",
> references to the legislation
> for offshore oil/gas installations, namely the England&Wales law known as
> Offshore Installations
> (Safety Case) Regulations 2005 (there are earlier versions). This law is
> *very prescriptive* as to
> what has to be included in an offshore installation Safety Case - you can
> read it at
> http://www.legislation.gov.uk/uksi/2005/3117/schedule/2/made
>
> One also finds references to the legislation for railways, namely the
> England&Wales law known as
> Railways (Safety Case) Regulations 2003 (there are earlier versions). This
> law is *prescriptive* as
> to what has to be included in a railway Safety Case - you can read it at
> http://www.legislation.gov.uk/uksi/2003/579/contents/made
>
> It should be evident that a Safety Case for an offshore installation and a
> Safety Case for a railway
> are *two different things*, because these two enterprises use different
> kit: pipes and pumps and
> drills versus rails and locomotives and carriages. So rather than
> > a term that has been defined for decades and is widely used to have a
> specific meaning
> there are at least two specific meanings in English law alone.
>
> It is also absurd to suggest that either of these Safety Cases is just
> some argument put together to
> fulfil a goal-oriented safety standard. First, they are not there to
> fulfil any standard, they are
> the law of England and Wales (I say again: standards aren't laws). Second,
> they are *prescriptive* -
> what shall be in a Safety Case designated as such in these industries is
> specified precisely in the
> law.
>
> So, sticking with Safety Cases with upper-case initial letters (or at
> least with mid-size initials),
> here is the definition of a Safety Case for the nuclear industry in
> Britain (notice this is not a
> law - the ONR surveys installations in Britain, and Britain has three
> separate legal regimes:
> England and Wales, Scotland, and Northern Ireland)
>
> [begin quote from: Purpose and Scope of Safety Cases NS-TAST-GD-051
> Revision 3]
>
> 5.      DEFINITION OF A NUCLEAR SAFETY CASE
> 5.1.    A nuclear safety case is a set of documents that describe the
> radiological hazards in terms of
> a facility or site and modes of operation (including potential undesired
> modes) and the measures
> that prevent or mitigate against harm being incurred. The safety case
> should provide a coherent
> demonstration that relevant standards have been met and that risks to
> persons have been reduced to
> as low as reasonably practicable (ALARP).
>
> [end quote]
>
> So there is, already, the third meaning. (To spell it out: neither of the
> previous two regulations
> "describe the radiological hazards" of a railway or an offshore
> installation.) It is obviously not
> as prescriptive as the first two, but it is still prescriptive.
>
> So let's move to a generic definition, as embodied in a standard which is,
> by the agreement forming
> the European Union, the standard governing safety related electronic
> systems for railway signalling
> in 28 European countries (EN 50129:2003, the document referenced by Les
> Chambers, in case readers
> haven't pursued the link):
>
> [begin quote EN 50129:2003 Section 3 Terms and Definitions]
>
> 3.1.49 safety case
> the documented demonstration that the product complies with the specified
> safety requirements
>
> [end quote]
>
> This is obviously a fourth definition, since it makes no reference to
> pipes&pumps and stuff, or to
> rails&locomotives and stuff, or to radiological hazards of a facility and
> stuff. Unlike the first
> three definitions, it is not prescriptive at all.
>
> Then there is my definition, which might as well be this: a rigorous
> argument, given in detail in a
> specified set of documents which include all the evidence used in the
> argument, that a system is
> acceptably safe, both when it is operating and when it is not operating.
>
> Notice that there is no requirement that all these documents have to
> reside in one place. It covers
> FAA and EASA certification practices and it is intended to cover the
> capital-letter Safety Case
> regimes above with a suitable interpretation of "acceptably safe".
>
> I'll leave it as an exercise to point out the differences between my
> definition and the previous four.
>
> Here's another, from
>
> https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/27585/SMP12v22final.pdf
>
> [begin quote UK MoD SMP 12 Version 22, citing Def Stan 00-56 Version 4]
>
> A structured argument, supported by a body of evidence that provides a
> compelling, comprehensible
> and valid case that a system is safe for a given application in a given
> environment.
>
> [end quote]
>
> So, unlike offshore installations or UK railways, a given MoD system might
> have a plethora of safety
> cases, if there are a plethora of applications in a plethora of
> environments. As well as depending
> on the possibly-varying meaning of the system property "safe". (I'd guess
> this is intended to mean
> something similar to my term "acceptably safe".)
>
> So much for "widely used to have a specific meaning"! More accurate would
> be to use the plural. It
> should also be evident that whether a safety case is narrowly prescriptive
> or more free in form
> depends on which industry, regulation and country you might be talking
> about.
>
> It's perfectly possible for there to be a good argument that the regime in
> Offshore Installations
> (Safety Case) Regulations 2003 in England & Wales "would not work" in the
> US oil/gas industry, as
> Nancy has indicated she believes. Such an argument would not obviously
> generalise to, say, FAA
> civilian aerospace certification documentation, which also forms a safety
> case - indeed people like
> me would say that that civilian aerospace safety cases are pretty good in
> general, while pointing
> out that they are by no means perfect, indeed it is possible to think that
> overall they are pretty
> good but specifically concerning avionics SW they are pretty poor.
>
> It is also possible to consider FAA safety cases de re free-form (a
> manufacturer can provide what
> evidence it wants to establish a certification goal set by negotiation
> between regulator and
> manufacturer, for example on certification of lithium-ion batteries for
> the Boeing 787) and de facto
> prescriptive (if manufacturers have provided such-and-such documentation
> for successful
> certification of kit X in the past, then it is pretty much de rigeur that
> a manufacturer will have
> to provide at least the equivalent of that for future kit-X certification
> proceedings. The
> certification requirements, and therefore the recertification
> documentation, after the Boeing 787
> battery fires were *modified* from what they had been, not thrown away and
> rewritten).
>
> PBL
>
> Prof. Peter Bernard Ladkin, Faculty of Technology, University of
> Bielefeld, 33594 Bielefeld, Germany
> Tel+msg +49 (0)521 880 7319  www.rvs.uni-bielefeld.de
>
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
>
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
>
>
>
> --
> Prof. Nancy Leveson
> Aeronautics and Astronautics and Engineering Systems
> MIT, Room 33-334
> 77 Massachusetts Ave.
> Cambridge, MA 02142
>
> Telephone: 617-258-0505
> Email: leveson at mit.edu
> URL: http://sunnyday.mit.edu
>
> #
>
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>
>

-- 
Prof. Nancy Leveson
Aeronautics and Astronautics and Engineering Systems
MIT, Room 33-334
77 Massachusetts Ave.
Cambridge, MA 02142

Telephone: 617-258-0505
Email: leveson at mit.edu
URL: http://sunnyday.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20140204/c5c0a1bc/attachment-0001.html>