[SystemSafety] Call for Submissions

Mike Ellims michael.ellims at tesco.net
Thu Aug 25 12:23:42 CEST 2016

Consider an app that is used to take photos of skin lesion and then send these to a doctor for diagnosis (there are such apps). If there is an error in the camera hardware or perhaps some sort of error in the software the adjusts the colour balance of the light (or a complete lack of said function) then this may result in misdiagnosis. A false positive being less of a problem than  a false negative as one keeps the concerned person away from the doctor the other sends them for a check-up.

Again another known issue, consider apps that are used by doctors to calculate drug doses. A software or data error in the app will could cause the wrong dosage to be administered (under/over == more/less) causing harm. This has been documented in the medical literature.

Likewise, if I remember correctly I believe that the Tesla (that company again :-) had issues with the app that allowed owners (and hackers) to remotely unlock their cars. I believe that there were no direct safety issues (or at least none stated) and it was fixed quickly once they knew about it but...


-----Original Message-----
From: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of Peter Bernard Ladkin
Sent: 25 August 2016 09:38
To: The System Safety List
Subject: [SystemSafety] Call for Submissions

Barrister Stephen Mason http://www.stephenmason.eu sent me a heads-up on the EU performing a consultation on the safety of apps. Non-embedded apps.


This seems prima facie weird. According to standard (engineering) definitions, such apps are not safety-related, period. It could be that the EU is looking for connections with safety which do not fit standard conceptions.

One model for a non-standard conception is ATC. (Although it is odd to call it "non-standard", I guess, since the system has been around way longer than the "standard" conceptions. Indeed, if one dates the initialisation of positive control to the 1956 Grand Canyon collision, it is three times as long.)

Let us consider airspace with 100% primary and secondary radar coverage. The data-gathering/-distribution/-display systems (let me call it real-time traffic display, RTTD) used by ATCOs obviously have a connection with safety in that an ATCO can take/inappropriate inappropriate actions on aircraft separation if the current traffic situation is not veridically displayed.

The safety of the airspace system could be defined in terms of the maintenance of appropriate separation of all participating aircraft, and more generally separation of participating aircraft from all other aircraft. Put briefly: no airproxes (however you might choose to define airprox).

Maintenance of safety is composed of three factors which form a causal chain: veridical RTTD; correct procedural actions by the ATCO based on the RTTD information; conformant execution of the agreed actions by flight crew.

Functional safety of the RTTD is guaranteed: RTTD paints pixels on screens and there are no known dangerous failures of painting pixels on screens. But it is equally clear that misleading information followed by nominally-appropriate action by an ATCO on that information followed by nominally-appropriate response by flight crew can result in loss of separation and thus an airprox.
In other words, a failure of dependability properties of the RTTD can by itself lead to a hazardous event; it does not need to be compounded with other failures.

The point is here that the behaviour of the other components of the causal chain is regulated, more or less completely.

So I guess the point of the consultation might be to elicit such causal chains in which the behaviour of an app in a causal chain can be similarly be sole cause.

But notice how the situation must be constructed. If the specific behaviour of the app can be mitigated by a change in behaviour of another agent in the chain, say by a human, then it can't be sole cause: an additional causal factor is that the mitigating behaviour did not occur. So unless the app executes in a context in which the behaviour of other actors is rigorously constrained, as in ATC, the app can't be sole cause of a hazardous event.

It'll be interesting to see what comes out of the consultation.


Prof. Peter Bernard Ladkin, Bielefeld, Germany MoreInCommon Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de

This email has been checked for viruses by Avast antivirus software.

More information about the systemsafety mailing list