[SystemSafety] Elephants, dinosaurs and integrating the VLA model

[Peter Bernard Ladkin]

On 2023-08-03 13:48 , Les Chambers wrote:
> Peter
> Your comment:
> "Martyn already observed on 2023-06-27 that there are legal requirements which
> constrain deployment of safety-related systems. That legal requirement in the
> UK and Australia is 77 years old. Your question seems to be suggesting that you
> somehow think it, and other constraints, might no longer apply. Well, they do.
> As Martyn said "AI doesn't change that.
> In the UK or Australia, developer and deployer must reduce risks ALARP.”
> 
> … is righteous … that is, if decreed by a king ruling by fiat (Latin for “let
> it be done”).

Yes, well, by far the largest part of Causalis Limited's clients has been lawyers working on 
accidents. You may think paying close attention to applicable law is "righteous". I think of it as 
common sense.

> The interesting question is, how does one PROVE (to an auditor or a judge) that
> one has reduced risks ALARP if one’s delivered system’s behaviour is initiated
> from a neural network? 

Yes, it is an interesting question. The current answer is that you show the neural network has 
limited scope for action and present the hazard analysis that shows that that limited scope leads 
only to acceptable risk, and that you can't easily do any better.

And if you can't do that, you're screwed (to coin a term).

> Case study: Elaine Herzberg, killed by a self-driving Uber in Tempe, Arizona in
> 2018. The system did not classify her as a pedestrian because she was crossing
> without a crosswalk; the neural net did not include consideration for
> jaywalking pedestrians.

What is the outcome of the case study?

I can't see what you are getting at with this line of thought.

PBL

Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de




-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230803/0517492a/attachment.sig>