[SystemSafety] Fwd: NYTimes: The Next Accident Awaits

Tue Feb 4 12:19:44 CET 2014

I have some issues with applying probabilistic based goals. I have come
across different hazard analyses (PHA, FMECA, etc.) which assign
frequencies. In these analyses, there are fields like "Hazard Description",
"Failure Mode", "Failure Cause", "Failure Effects", followed by a
"Frequency", "Severity" and "Risk" field. In some cases it is not even
clear whether the "Frequency" has been assigned to the consequence
("Failure Effect") or the hazard itself. I am aware that a hazard does not
always lead to a risk, however, no distinction between the frequency of the
hazard and the risk is made in the analysis.

Suppose the frequency is assigned to the risk identified as a consequence
of the hazard. A frequency must be assigned to the risk/consequence, a
mitigation measure/barrier proposed, and the frequency/severity/risk is
analysed again taking into account the mitigation measures (at least in my
experience). This is considered documentary evidence that a risk has been
mitigated, usually to be included in a Safety Case.

My issue with this approach is that I have yet to come across numerical
data which support the frequencies assigned. You can establish risk
acceptance criteria for the risk, but I am referring to the "estimated"
numerical frequency of the risk itself. In the standard which I work with,
the EN 50126, frequencies are defined as, for example*, "Frequent. Likely
to occur frequently. The hazard will be continually experienced"*. Or
"*Probable.
Will occur several times. The hazard can be expected to occur
often."*Often these categories are accompanied by "typical" numerical
values, which
are an estimation of the occurrence of the risk, assigned by the people
preparing the hazard analysis. Where are the data which really show that
the frequency you have "estimated" for your risk is actually representative
of the real frequency? You can set probabilistic based goals, but how can
you compare with those goals if you don't estimate the frequency of your
risk correctly? What is the numerical definition of "*Frequent*", "
*Probable*" etc., supported by evidence? Thus the frequency is no longer
"measurable".  Especially in industries with "teething problems". As Philip
Koopman mentioned, there has been an electromechanical à software
transition. Many of the risks identified may be new. Furthermore,
justifying the risk reduction after applying the mitigation measures seems
to be even more difficult. How do you justify that you lowered the
frequency to "Remote", rather than "Improbable"? or "Incredible" rather
than "Improbable"? The IEC 61508 has methods for calculating the frequency
of dangerous failure which are suitable for established components like
sensors, but they may not be so suitable for risks  in younger industries.
I had considered starting a new thread, though it seems to me to be part of
the same topic.

Regards,

Myriam.

2014-02-04 Nancy Leveson <leveson.nancy8 at gmail.com>:

> There are more alternatives than just checklists and "objectives." And
> usually more than one may be used, especially the use of checklists plus
> other methods. Here is how I characterized the alternatives in my paper on
> safety cases:
>
> *Types of Regulation*
>
>      Certification methods differ greatly among industries and countries.
> Approaches commonly used can be broken into two general types, which
> determine the type of evidence used in the certification process:
>
>    1. *Prescriptive*: Standards or guidelines for product features or
>    development processes are provided that are used to determine whether a
>    system should be certified.
>
>
>     1. *Product*: Specific design features are required, which may be (a)
>       specific designs or (b) more general features such as fail-safe design or
>       the use of protection systems.
>
>                                            i.     *Specific designs and
> features* provide a way to encode and pass on knowledge about past
> experience and lessons learned from past accidents. In some industries,
> practitioners are licensed based on their knowledge of the standards or
> codes of practice. An example is the existence of electrical codes based on
> past experience with various designs. For software, some completeness
> criteria for requirements have been identified [Leveson, 1995] as well as
> specific design features [Leveson, 2012] based on common flaws leading to
> many accidents in the past. Certification then becomes the responsibility
> of the licensed practitioner, who can lose their license if they fail to
> follow the standards. Organizations may also be established that produce
> standards and provide certification, such as the UL rating. It is difficult
> to fathom any argument that such encoded knowledge should not be included
> in any certification effort. Requiring reinvention of this past experience
> for every project would be prohibitively costly and potentially incomplete
> and error prone without any clear advantage.
>
>                                          ii.     Different industries
> face different safety problems and therefore the *general approach to
> safe design* may differ among them. For example, commercial aviation has
> created various types of fail-safe techniques used to protect against
> component failures [Follensbee]. Nuclear power, because of differences in
> the problem, has traditionally used defense in depth and protection
> systems. For software, such features might include the use of exception
> handling, checking for out-of-range variables, and designing to reduce the
> potential for human error when interacting with the software. Certification
> is usually provided by inspection that the design features provided are
> effective and implemented properly.
>
>
>
>     1. *Process*: Here the standards specify the process to be used in
>       producing the product or system or in operating it (e.g., maintenance or
>       change procedures) rather than specific design features of the product or
>       system itself. Assurance is based on whether the process was followed and,
>       sometimes, on the quality of the process or its artifacts. The process
>       requirements may specify
>
>                                                      i.     General
> product or system *development processes and their artifacts*, such as
> requirements specifications, test plans, reviews, analyses to be performed
> and documentation produced (e.g., DO-178B) or
>
>                                                    ii.     The *process
> to be used in the safety engineering* of the system and not the general
> development process used for the product (e.g., MIL-STD-882). Only the
> safety engineering process is specified, not the general development
> process, which is up to the individual system developers.
>
>
>
>    1. *Performance-based or goal-setting approaches* focus on desired,
>    measurable outcomes, rather than required product features or prescriptive
>    processes, techniques, or procedures. The certification authority specifies
>    a threshold of acceptable performance and often but not always a means for
>    assuring that the threshold has been met. Basically, the standards set a
>    goal, which may be a risk target, and usually it is up to the assurer to
>    decide how to accomplish that goal. Performance-based regulation specifies
>    defined results without specific direction regarding how those results are
>    to be obtained. An example is a requirement that an aircraft navigation
>    system must be able to estimate its position to within a circle with a
>    radius of 10 nautical miles with some specified probability or that for new
>    aircraft in-trail procedure (ITP) equipment "The likelihood that the ITP
>    equipment provides undetected erroneous information about accuracy and
>    integrity levels of own data shall be less than 1E-3 per flight hour"
>    [RTCA, 2008].
>
> Nancy
>
>
> On Tue, Feb 4, 2014 at 5:40 AM, SPRIGGS, John J <John.SPRIGGS at nats.co.uk>wrote:
>
>>  As one of "the grey beards" someone mentioned several posts ago, I
>> first encountered goal-based regulation about 2001, when my Customers'
>> Regulator said they were about to introduce it.  Customers had asked for
>> Safety Cases long before that.  The argument would be along the lines of
>> "You told us to do this, here is the evidence we did it; we also did this
>> other stuff to address the hazards we identified along the way; therefore
>> we consider it safe enough for the purpose you stated."
>>
>> So should the debate be about whether Regulation should be based on
>> checklists or objectives, rather than whether you need a safety case or
>> some other assurance document?
>>
>>
>>
>> John Spriggs
>>
>> Head of System Integrity @ NATS
>>
>> .
>>
>> .
>>
>> .
>>
>>
>>
>>
>> ------------------------------
>> If you are not the intended recipient, please notify our Help Desk at
>> Email Information.Solutions at nats.co.uk immediately. You should not copy
>> or use this email or attachment(s) for any purpose nor disclose their
>> contents to any other person.
>>
>> NATS computer systems may be monitored and communications carried on them
>> recorded, to secure the effective operation of the system.
>>
>> Please note that neither NATS nor the sender accepts any responsibility
>> for viruses or any losses caused as a result of viruses and it is your
>> responsibility to scan or otherwise check this email and any attachments.
>>
>> NATS means NATS (En Route) plc (company number: 4129273), NATS (Services)
>> Ltd (company number 4129270), NATSNAV Ltd (company number: 4164590) or NATS
>> Ltd (company number 3155567) or NATS Holdings Ltd (company number 4138218).
>> All companies are registered in England and their registered office is at
>> 4000 Parkway, Whiteley, Fareham, Hampshire, PO15 7FL.
>> ------------------------------
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>>
>>
>
>
> --
> Prof. Nancy Leveson
> Aeronautics and Astronautics and Engineering Systems
> MIT, Room 33-334
> 77 Massachusetts Ave.
> Cambridge, MA 02142
>
> Telephone: 617-258-0505
> Email: leveson at mit.edu
> URL: http://sunnyday.mit.edu
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20140204/357d4f58/attachment-0001.html>