[SystemSafety] Therac-25 redux

[Les Chambers]

Thanks Peter

Vita brevis, ars longa.

I guess Hippocrates was right. In a single lifetime no one person will ever
accumulate the knowledge of what could go wrong in any domain; a single life
is too short. Collective eternal memory requires collaboration (across
generations, across millennia) and an accessible body of knowledge. Don't
say it can't be done, if Hippocrates (460 – 370 BCE) can still be quoted
today so can modern wisdom.
For the first time ever the web has made eternal collaborative memory
possible. When I published this book on Amazon:
http://www.amazon.com.au/How-Lucky-Was-Rex-Kimlin-ebook/dp/B008RKRXS2
... it occurred to me that it will never go "out of print"; the author, Rex
Kimlin is in his nineties and won't be with us for much longer. But the
thoughts of a man who risked his life 35 times with a 50 percent probability
of death will be with us for as long as Amazon exists (is Bezos immortal?).
The same is true for Harold Thimbleby's book (Press On). It looks great by
the way, anyone extolling the virtues of state engines for interactive
design has got to be a righteous dude.
Well written books on system safety are a great thing. But I think we need
to take publishing-what-could-go-wrong a step further. And that is to make
the kernels of wisdom present in all these books more accessible in an
abbreviated form, expressed as an ontology. For example, if pressed, I could
probably reduce this tome:
http://www.systemsengineeringblog.com/deus_ex_machina/ to and A4 page of
managerial bad behaviour patterns that lead to disaster; how to recognise
them; how to overcome. 
Published on the Web this kind of thing would be invaluable to anyone doing
a hazard analysis. Ph.D. thesis material perhaps? How would you structure
such a thing? A wiki for disaster. How would you triumph over the guardians
of polite capitalism and move beyond the pettiness of "I won't divulge my
hazards in case someone steals my stuff." It seems a shame that the outcome
of collective memory loss and ignorance should be the death of innocents.
By the way, Caesar Augustus died 2000 years ago tomorrow (Tue 19 August).
We have him to thank for the month of August. 
Let us not forget.
Cheers
Les

-----Original Message-----
From: systemsafety-bounces at lists.techfak.uni-bielefeld.de
[mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of
Peter Bernard Ladkin
Sent: Saturday, August 16, 2014 5:13 PM
To: systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] Therac-25 redux



On 2014-08-16 06:10 , Les Chambers wrote:
> A quick search of the Internet did not reveal any publication of
drug-infusion pump hazards. Is
> anyone aware of same?

Yes we are (I much less than some others here). Vulnerabilities with medical
devices, especially
implantable medical devices, are a big thing. Harold Thimbleby at Swansea
has been working on it for
a couple decades. He's primarily an HMI guy but we wrote a couple papers on
security and safety. He
has an award-winning book on interface design with MIT Press called Press
On.
http://www.cs.swan.ac.uk/~csharold/

Ross Anderson at Cambridge is aware of the issues with medical device
security, but works primarily
in other areas. He was at Black Hat this year, where the IOActive stuff on
the Cobham kit was
presented. He does have some strong opinions on the state of the practice in
medical-device security

Barnaby Jack was one of the best known (that is, notorious) security
thespians. I understand he was
about to demo defibrillator and infusion-pump vulnerabilites at Black Hat
last year when he
overdosed himself on recreational and other drugs a week before. He has a
Wikipedia page, which one
can be sure was not written by him :-)

You have to be somewhat careful of the "security theatre" surrounding
medical-device
vulnerabilities. I am told that patient welfare is not being well served by
the current addiction to
media exposure. Indeed, it is one of the three topics in the paper I wrote a
week ago for the
upcoming SSS in February in Bristol http://www.safety-club.org.uk/e300 .
(One of the others is, by
request, MH 17. It is about how one might do security risk assessment. It
turns out to be different
in some crucial ways from safety risk assessment.)

There are indeed stories to be told, and recently I have been reading some.
Neither Harold nor Ross
is on this list, but one of our lurkers is a renowned expert on
medical-device safety and security.

One of the big problems, not well served by security theatre, is that some
of the implantable kit
was designed and implanted quite a while ago, before people paid that much
attention to the kind of
antics security thespians can get up to nowadays. But fixing it, that is,
updating a device,
requires more surgery, which is not without risk and of course considerable
inconvenience to the
patient. That has to be balanced against the chances that some jerk behind
you in the line at
Starbucks will reprogramme your defibrillator with a phone.

I would imagine that one of the reasons this topic is hitting the press now
is, as The Economist
hinted, the US FDA appears to be embarking on a push to get this all sorted.

> This brings me to my point: wouldn't it be great if we had a readily
accessible ontology of hazards
> for various application domains. It's an obvious idea. Is anyone aware of
discussions along these
> lines? 

I think, from what I understand, that that's part of the FDA plan. It's
certainly part of general EU
planning: http://www.enisa.europa.eu Ross Anderson, Rainer Böhme, Richard
Clayton and Tyler Moore
have a report "Security Economics and the Internal Market" for ENISA, which
says as one of its first
recommendations that there should be vulnerability databases with compulsory
notification requirements.

> "Open source" hazard ontologies would solve the problem of corporate
memory loss, amnesia and
> denial. 

Yes, but I doubt there is any chance. Too much proprietary information is
involved for any effective
vulnerability catalogue to be public.

PBL

Prof. Peter Bernard Ladkin, Faculty of Technology, University of Bielefeld,
33594 Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs.uni-bielefeld.de




_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE