[SystemSafety] NYTimes: The Next Accident Awaits

[Andrew Rae]

This may be an appropriate time to mention a paper John McDermid and I
wrote in 2012,
"Goal Based Safety Standards: Promises and Pitfalls". The title was written
and promised before the paper, so it
doesn't quite capture the fact that it is mainly about the question of
whether it is possible, even in principle, to empirically determine
which form of regulation works best.

Whilst the authors are from the "other side" of the goal-based debate, it
makes the same epistemological point as Nancy was making. We shouldn't be
making sweeping claims about what works unless we have evidence to back
them up. At the level of national (or even industry-by-industry) regulation
the
complexity of confounding factors is ridiculous. It's a hard enough
empirical problem to determine who is actually safer let alone why.

I think the confirmation bias issue (and review in general) is an area
where we could do some effective experimental work.
  What types of errors are reviewers good or bad at identifying?
  Does making the safety argument explicit help or hinder a reviewer in
finding weaknesses?
  To what extent can you "prime" a reviewer to believe a system is probably
safe, and how does this change review performance?
  Which types of evidence actually help us tell the difference between a
safe and unsafe system?

There's a management science paper "Resolving scientific disputes by the
joint design of crucial experiments by the antagonists: Application to the
Erez-Latham dispute regarding participation in goal setting" which suggests
that where you have entrenched scientific disputes one way forward is to
design an experiment together.
Whilst some of the really big questions like "which form of regulation
works best, when" are beyond our resources to answer, it is probably worth
thinking about what experiments (or more likely non-experiment empirical
studies) would reveal the answers to disputes in system safety. If nothing
else it might help find smaller questions (e.g. those I listed above) where
we could realistically reach agreement based on evidence.

[My own provisional view on safety cases - If they are done properly, there
is no reason to think that they shouldn't be better than prescriptive
regulation, because even when using prescription you still have to address
the problem of suitability and applicability of the regulation, and there
is no mechanism to capture this. HOWEVER, I am not at all convinced that
any industry is consistently using safety cases properly (rail is the one
possible exception, but there is a very heavy historical and regulatory
background to the type of argument and evidence used inside the safety case
framework). If my car is super-safe when driven under 40mph, but everyone
always drives at 50mph, there's a point where I have to stop insisting it
is a safe car.]


My system safety podcast: http://disastercast.co.uk
My phone number: +44 (0) 7783 446 814
University of York disclaimer:
http://www.york.ac.uk/docs/disclaimer/email.htm

My system safety podcast: http://disastercast.co.uk
My phone number: +44 (0) 7783 446 814
University of York disclaimer:
http://www.york.ac.uk/docs/disclaimer/email.htm


On 2 February 2014 11:30, Nancy Leveson <leveson.nancy8 at gmail.com> wrote:

> I served as an expert consultant to the Presidential Oil Spill Commission
> after Deepwater Horizon and helped write the report. Many people at that
> time were suggesting that all our troubles would be solved by adopting
> safety cases. As a result, I started studying this topic in depth, read
> everything I could find written on it, and in the end wrote a paper against
> the use of a safety case regulatory regime in the U.S. Here are some of my
> arguments (see the entire paper for details):
>
> 1. Confirmation Bias: Confirmation bias (a well established psychological
> principle) leads to incorrect safety cases (as have been most of the safety
> cases that I have seen published). And reviewers suffer from the same type
> of confirmation bias as those making up the cases. Without some way of
> combating confirmation bias, letting people make up arguments for safety or
> requiring certifiers to evaluate each argument individually is not going to
> be as effective as prescriptive regulation based on historical precedent.
>
> 2. Impractical Expertise Requirements on the part of Regulators: It is
> impossible for regulators to be an expert on every type of argument and
> analysis method that could possibly be used by an applicant. It is much
> more difficult and more error prone to have to evaluate any possible
> argument given. Where will such experts come from? If they exist, will they
> really want to work for government wages (at least in the US)? I don't know
> many people who could do this job well, including myself. So are arguments
> simply accepted because they sound good?
>
> At a meeting last year, I spoke informally with a European regulator who
> argued that he could not regulate without the use of PRA. His argument was
> that the systems in his industry were becoming so complex that the
> regulators could not possibly understand the details of the systems they
> were certifying. So they accepted probabilistic arguments by applicants
> that performance targets would be met. I asked him how the regulators could
> possibly know if the PRA results were correct or even reasonable if they
> did not understand the designs that were being analyzed? He had no answer
> for this question. In aviation, for example, it would be impossible for any
> regulator to understand the details of the design of the entire plane in
> order to follow an argument for why that design is safe. In addition, most
> of these details are proprietary and therefore safety cases would not be
> able to be open to the public or to any independent evaluation.
>
> 3. Impractical Resource Requirements:  The safety case approach requires
> not only more expertise on the part of regulators, but more resources. The
> number of government resources required to apply such a regulatory regime
> adequately are much more than would be practical in many countries,
> including the U.S. For example in off-shore oil drilling, the UK and Norway
> employ a large number of highly educated personnel and technical
> specialists to perform audits, inspections and review required documents.
> The UK has about an equal number of off-shore oil rig inspectors as they
> have off-shore oil rigs. In Norway, the PSA has approximately 160
> employees, of which approximately 100 perform compliance and audit-related
> tasks regulating 105 offshore installations. Each of these 100 employees
> has a postgraduate (Master's degree) or equivalent level of training, in
> one of more areas of expertise, including drilling, petroleum engineering,
> structural engineering, and reliability engineering. In contrast, in the
> U.S., the Bureau of Safety and Environmental Enforcement (BSEE) and the
> U.S. Coast Guard share approximately 60 billeted offshore inspectors over
> 3,500 offshore installations. We would never be able to hire the number of
> people or put in the resources that the British and Norwegians do. It would
> simply devolve into lack of any adequate regulatory oversight by U.S.
> agencies due to lack of adequate personnel. Personnel requirements are less
> for prescriptive regulation.
>
> 4. Does it work? Is it better? There have been few objective studies
> conducted on the impact of the safety case regulatory approach on safety
> performance vs. other approaches. It would be nice before we engage in
> "proof by vigorous handwaving and strong advocacy" if people would collect
> scientific evidence of the superiority of the safety case approach over
> others. Proponents have not done so. Note, however, that the industries
> with the best accident statistics (such as civil aviation) do not use
> safety cases but rather use prescriptive regulation. So a scientific,
> comparative evaluation should be made by those advocating this approach as
> well as ways to overcome the three practical difficulties listed above.
> Just because it sounds good or is different than what we do now is not
> enough.
>
> Nancy
>
> On Sun, Feb 2, 2014 at 4:53 AM, Tracy White <tracyinoz at mac.com> wrote:
>
>> I have found through personal experience that people with a
>> 'certification' pedigree struggle with the concept if a safety case ...
>> this is particularly true in defence. Where people have come from the
>> prescriptive world which calls for completion if tasks x,y, z; their safety
>> case is then: it's safe because we did x,y,z. This approach completely
>> fails to justify or explain why x,y,z is appropriate or sufficient for
>> their particular project.
>>
>> I do not believe that 'safety cases' provides a free for all as, in the
>> absence of a suitable alternative,  the same prescriptive sources will
>> feature as technical safety measures. But what the safety case should bring
>> to the table is a requirement to satisfy a claim as to why these measures
>> (or any others) are sufficient, appropriate, applicable, relevant etc.
>> something that prescription fails to do.
>>
>> Regards, Tracy
>>
>> On 2 Feb 2014, at 19:05, Nancy Leveson <leveson.nancy8 at gmail.com> wrote:
>>
>> I don't think that anyone is implying that the safety case "replaces some
>> form of regulation". But it implies a particular form of regulation,
>> usually performance-based rather than prescriptive. Thus ARP 4751 in
>> aviation and MIL-STD-882 in defense, are not safety case regimes because
>> there are specific procedures that must be followed to be certified. The
>> applicant does not get to determine what type of argument they make.
>>
>> Nancy
>>
>>
>> On Sat, Feb 1, 2014 at 7:43 PM, Tracy White <tracyinoz at mac.com> wrote:
>>
>>> I am slightly confused and a little perturbed by an argument that a
>>> 'safety case' in someway replaces any regulatory control (or government
>>> interference). Even more that a safety case would not include a subclaim to
>>> have conducted a 'rigorous hazard analysis' program ... or to have applied
>>> appropriate 'procedures and standards'.
>>>
>>> Anybody who thinks that 'safety cases' in anyway replaces some form of
>>> regulation is ignorant of its purpose. I work in a regulatory environment
>>> and the 'safety case' is the primary communications  medium with that
>>> regulator,  elements of which will talk to hazard identification and
>>> compliance with standards and codes considered representative of
>>> engineering 'good practice'. I would agree that there are good and bad
>>> safety cases and I think that 'industries that do not 'have a good
>>> historical culture in terms of safety' are as ignorant of purpose of the
>>> safety cases as they of the need for safety in general.
>>>
>>> Regards, Tracy
>>>
>>> On Feb 01, 2014, at 12:48 AM, Nancy Leveson <leveson.nancy8 at gmail.com>
>>> wrote:
>>>
>>> It is very difficult to characterize the U.S. In general, the country is
>>> so physically large that there are extreme differences in culture and
>>> politics (generally but not always physically bounded). Much of the central
>>> government in the US and European worlds seem to be moving toward
>>> libertarianism, but I am probably mischaracterizing Europe based on biased
>>> news reports. The individual U.S. states show extreme differences. At the
>>> extremes, Texas and California may as well be in different worlds, let
>>> alone countries when it comes to safety regulations (and lots of other
>>> things irrelevant to this list). There are also such different cultures in
>>> different industries that it is difficult to make general statements.
>>> Mining and civil aviation are examples of such extremes.
>>>
>>> But I will make one general statement that is only my personal
>>> experience. Because of my paper arguing against safety cases, I am getting
>>> many calls from government employees and company lawyers as well as
>>> individual engineers. Some of the companies pushing the "safety case" in
>>> the U.S. are those who don't want any government interference and see the
>>> safety case as a way to get around the rigorous procedural standards that
>>> now exist here in many industries. They seem to feel that they will be able
>>> to get rid of the procedures and standards that exist now and can write
>>> anything they want in a safety case and therefore save money and time in
>>> the rigorous hazard analysis now widely required while using any design
>>> features they want. These are primarily in industries that do not have a
>>> good historical culture in terms of safety.
>>>
>>> Nancy.
>>>
>>>
>>> On Fri, Jan 31, 2014 at 4:08 AM, RICQUE Bertrand (SAGEM DEFENSE
>>> SECURITE) <bertrand.ricque at sagem.com> wrote:
>>>
>>>> Hi Nancy,
>>>>
>>>>
>>>>
>>>> Concerning France you are right, and in that case I think that the
>>>> cultural aspect dominates. There is no safety culture in the population as
>>>> in UK, as acknowledged after AZF accident. The risk stops at the fence of
>>>> the plant and you can safely build your house on the other side ... The
>>>> regulations have changed since but not the cultures. The safety engineers
>>>> concerned by the new regulations live a nightmare as the choices are more
>>>> or less, dismantle the plant versus dismantle the town ... I think that the
>>>> safety cultures have more impact on the final result than the competence of
>>>> the safety community.
>>>>
>>>>
>>>>
>>>> Bertrand Ricque
>>>>
>>>> Program Manager
>>>>
>>>> Optronics and Defence Division
>>>>
>>>> Sights Program
>>>>
>>>> Mob : +33 6 87 47 84 64
>>>>
>>>> Tel : +33 1 59 11 96 82
>>>>
>>>> Bertrand.ricque at sagem.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* systemsafety-bounces at lists.techfak.uni-bielefeld.de [mailto:
>>>> systemsafety-bounces at lists.techfak.uni-bielefeld.de] *On Behalf Of *Nancy
>>>> Leveson
>>>> *Sent:* Thursday, January 30, 2014 8:59 PM
>>>> *To:* systemsafety at lists.techfak.uni-bielefeld.de
>>>> *Subject:* Re: [SystemSafety] NYTimes: The Next Accident Awaits
>>>>
>>>>
>>>>
>>>> It would be nice to actually introduce some data into the discussions
>>>> on this list. First, although it is very true that the U.K. has excellent
>>>> comparative occupational safety statistics, this exceptional performance
>>>> predated safety cases by at least 100 years and is as much a cultural
>>>> artifact of the U.K. as any current practices. While the rest of the world
>>>> was suffering the results of steam engine explosions in the late 1800s, for
>>>> example, Great Britain was the first to implement measures to reduce them.
>>>> (I wrote a paper on this once if anyone is interested.) Although the
>>>> British citizens on this list know more about the history of the UK HSE, I
>>>> believe they were the first country to require companies to have safety
>>>> policies, etc., after the Flixborough explosion. Safety cases, I believe,
>>>> came into being only after the more recent Piper Alpha explosion.
>>>>
>>>>
>>>>
>>>> Trying to tie accident rates in different countries to particular ways
>>>> of regulating safety is dicey at best. First, there are significant
>>>> differences between the engineering, agricultural, industry, and service
>>>> rates of accidents in countries, often related to technical differences.
>>>> Some have high agricultural accident rates but low service accident rates.
>>>> For example, accident rates are going to be very different in a country
>>>> with high tech agricultural techniques compared to those still plowing
>>>> fields with a pair of oxen. Politics plays an even more important role. For
>>>> example, western countries often put very dangerous processes and plants in
>>>> third world countries or governments in these countries do not have laws
>>>> that require manufacturers to use even minimal safety practices in
>>>> manufacturing, for example, and they will not as long as they need the
>>>> revenue and jobs. The safety culture in these countries will not change
>>>> magically by using one type of regulatory regime.
>>>>
>>>>
>>>>
>>>> Note also, that there are vast differences in industries. Those with
>>>> the very safest records, such as the U.S. SUBSAFE program, do not use
>>>> safety cases. (And they have managed to have an incredible safety record
>>>> despite being in the U.S. :-)). If we want to compare the effectiveness of
>>>> different regulatory regimes, then we need to provide scientific
>>>> evaluations and not just misuse statistics (which may involve factors that
>>>> have nothing to do with the actual regulatory regime used).
>>>>
>>>>
>>>>
>>>> Also, as Michael Holloway noted, culture differences will make
>>>> different types of regulation more or less different in different countries
>>>> and industries.
>>>>
>>>>
>>>>
>>>> Finally, I would like to point out to those who are making some
>>>> national comparisons and putting down the U.S. in comparison with France,
>>>> for example, that the fatal occupational accident rate in the U.S. is less
>>>> than that of France. Perhaps we can avoid mixing politics and chauvinism
>>>> with science on this list.
>>>>
>>>>
>>>>
>>>> Nancy
>>>>
>>>>
>>>>
>>>> On Thu, Jan 30, 2014 at 8:50 AM, Martyn Thomas <
>>>> martyn at thomas-associates.co.uk> wrote:
>>>>
>>>> I'm a non-exec Director at the UK's Health and Safety Laboratory (
>>>> www.hsl.gov.uk). We carry out the basic research that underpins the
>>>> UK's regulation of occupational health and safety, ranging from reducing
>>>> accidents on construction sites and improving the tethering of loads on
>>>> lorries, through to reproducing and analysing major explosions (such as
>>>> Buncefield - http://www.buncefieldinvestigation.gov.uk/) and
>>>> destruction-testing the physical integrity of tankers and rolling-stock.
>>>>
>>>> We also undertake commercial work that uses our unusual experimental
>>>> and analysis capabilities and very strong science base.
>>>>
>>>> The UK is unusual in having a goal-based, safety-case regulatory regime
>>>> and a regulator (HSE) with its own expert research establishment (HSL). We
>>>> are getting an increasing number of approaches from Governments in the Far
>>>> and Middle East who see the UK's good performance in occupational Health
>>>> and Safety and who want to investigate setting up similar goal-based
>>>> regulation.
>>>>
>>>> Maybe there is something in the HSE/HSL approach that the US chemical
>>>> industry could benefit from.
>>>>
>>>> Regards
>>>>
>>>> Martyn
>>>> Martyn Thomas CBE FREng
>>>>
>>>>
>>>>
>>>>
>>>> On 29/01/2014 22:05, Peter Bernard Ladkin wrote:
>>>>
>>>> A worthy opinion piece from the Chair of the US Chemical Safety Board. Note his suggestion that identifying hazards and mitigation is just well-established best practice. I can say from experience that it is not yet in Europe in all industries with safety aspects, even though he holds Europe up as having a factor of three fewer chemical accidents as the US.
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> The System Safety Mailing List
>>>> systemsafety at TechFak.Uni-Bielefeld.DE
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Prof. Nancy Leveson
>>>> Aeronautics and Astronautics and Engineering Systems
>>>> MIT, Room 33-334
>>>> 77 Massachusetts Ave.
>>>> Cambridge, MA 02142
>>>>
>>>> Telephone: 617-258-0505
>>>> Email: leveson at mit.edu
>>>> URL: http://sunnyday.mit.edu
>>>>
>>>> #
>>>> " Ce courriel et les documents qui lui sont joints peuvent contenir des
>>>> informations confidentielles, être soumis aux règlementations relatives au
>>>> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
>>>> pas destinés, nous vous signalons qu'il est strictement interdit de les
>>>> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
>>>> soit le contenu. Toute exportation ou réexportation non autorisée est
>>>> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
>>>> l'expéditeur et de supprimer immédiatement de votre système informatique ce
>>>> courriel ainsi que tous les documents qui y sont attachés."
>>>>
>>>> ******
>>>> " This e-mail and any attached documents may contain confidential or
>>>> proprietary information and may be subject to export control laws and
>>>> regulations. If you are not the intended recipient, you are notified that
>>>> any dissemination, copying of this e-mail and any attachments thereto or
>>>> use of their contents by any means whatsoever is strictly prohibited.
>>>> Unauthorized export or re-export is prohibited. If you have received this
>>>> e-mail in error, please advise the sender immediately and delete this
>>>> e-mail and all attached documents from your computer system."
>>>> #
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Prof. Nancy Leveson
>>> Aeronautics and Astronautics and Engineering Systems
>>> MIT, Room 33-334
>>> 77 Massachusetts Ave.
>>> Cambridge, MA 02142
>>>
>>> Telephone: 617-258-0505
>>> Email: leveson at mit.edu
>>> URL: http://sunnyday.mit.edu
>>> _______________________________________________
>>> The System Safety Mailing List
>>> systemsafety at TechFak.Uni-Bielefeld.DE
>>>
>>>
>>> _______________________________________________
>>> The System Safety Mailing List
>>> systemsafety at TechFak.Uni-Bielefeld.DE
>>>
>>>
>>
>>
>> --
>> Prof. Nancy Leveson
>> Aeronautics and Astronautics and Engineering Systems
>> MIT, Room 33-334
>> 77 Massachusetts Ave.
>> Cambridge, MA 02142
>>
>> Telephone: 617-258-0505
>> Email: leveson at mit.edu
>> URL: http://sunnyday.mit.edu
>>
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>>
>>
>
>
> --
> Prof. Nancy Leveson
> Aeronautics and Astronautics and Engineering Systems
> MIT, Room 33-334
> 77 Massachusetts Ave.
> Cambridge, MA 02142
>
> Telephone: 617-258-0505
> Email: leveson at mit.edu
> URL: http://sunnyday.mit.edu
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20140202/5321e1b0/attachment-0001.html>