[SystemSafety] Elephants, dinosaurs and integrating the VLA model

[Steve Tockey]

Les wrote:

“Case study: Elaine Herzberg, killed by a self-driving Uber in Tempe, Arizona in
2018. The system did not classify her as a pedestrian because she was crossing
without a crosswalk; the neural net did not include consideration for
jaywalking pedestrians.”

The whole story is a bit more complicated than that. Here is a good summary of the NTSB report:

<https://arstechnica.com/cars/2019/11/how-terrible-software-design-decisions-led-to-ubers-deadly-2018-crash/>
[Screen-Shot-2019-11-06-at-4.10.29-PM-760x380.png]
How terrible software design decisions led to Uber’s deadly 2018 crash<https://arstechnica.com/cars/2019/11/how-terrible-software-design-decisions-led-to-ubers-deadly-2018-crash/>
arstechnica.com<https://arstechnica.com/cars/2019/11/how-terrible-software-design-decisions-led-to-ubers-deadly-2018-crash/>


“Imagine yourself as an expert witness supporting Tesla in a similar situation.
What section, subsection or footnote of IEC 61508 or ISO 26262 - or other
standard - would you cite to prove Elon had applied best practice in his
development life cycle?”

I find it hard to believe (nah, impossible actually) that I would ever agree to be an expert witness on the side of someone like Tesla. Rather, I think being an expert witness on the other side would be far easier. It would probably be almost trivial to prove Elon had NOT applied anything remotely similar to best practice in his development life cycle.


— steve




On Aug 3, 2023, at 7:48 PM, Les Chambers <les at chambers.com.au> wrote:

Peter
Your comment:
"Martyn already observed on 2023-06-27 that there are legal requirements which
constrain deployment of safety-related systems. That legal requirement in the
UK and Australia is 77 years old. Your question seems to be suggesting that you
somehow think it, and other constraints, might no longer apply. Well, they do.
As Martyn said "AI doesn't change that.
In the UK or Australia, developer and deployer must reduce risks ALARP."

. is righteous . that is, if decreed by a king ruling by fiat (Latin for "let
it be done").

Legal requirements are one thing, usually coming into play to the right of
"bang"; keeping the public safe in the first place is another more important
issue.
The interesting question is, how does one PROVE (to an auditor or a judge) that
one has reduced risks ALARP if one's delivered system's behaviour is initiated
from a neural network? A dataset that cannot be interpreted, verified or
validated thoroughly in process, and that changes after delivery. AI
aficionados admit they don't understand why NNs can work so well or fail so
unpredictably.
Witness: https://dawnproject.com/

Case study: Elaine Herzberg, killed by a self-driving Uber in Tempe, Arizona in
2018. The system did not classify her as a pedestrian because she was crossing
without a crosswalk; the neural net did not include consideration for
jaywalking pedestrians.

These systems are famous for not knowing what they don't know and imposing
their ignorance on the real world. Hannah Arendt was prescient: "It's not so
much that our models are false, but that they might become true"

Imagine yourself as an expert witness supporting Tesla in a similar situation.
What section, subsection or footnote of IEC 61508 or ISO 26262 - or other
standard - would you cite to prove Elon had applied best practice in his
development life cycle?

Or, if you cannot pony up, would you agree that these standards are no longer
fit for purpose in regulating the development of AI-integrated Safety-Critical
systems?

And furthermore, please explain the purpose of these standards, if they cannot
be instrumental in stopping the murder for money currently occurring on US
roads?

Les

PS: I note that Tesla's full self-driving (FSD) feature is available in the UK
as well as the US. It is not available in Australia or Germany.

---------------------------
On 2023-08-03 02:32 , Les Chambers wrote:

Can anyone on this list refer me to where in the standards one can obtain
guidance on how to engineer such a system safely?

That seems to be a question with a completely obvious answer.

Martyn already observed on 2023-06-27 that there are legal requirements which
constrain deployment
of safety-related systems. That legal requirement in the UK and Australia is
77 years old. Your
question seems to be suggesting that you somehow think it, and other
constraints, might no longer
apply. Well, they do. As Martyn said "AI doesn't change that."

In the UK or Australia, developer and deployer must reduce risks ALARP.

How do you go about engineering any system such that risks are reduced ALARP,
say in the UK? You
follow sector-specific functional safety standards if there are some, as well
as the engineering
functional safety standard for E/E/PE systems, which is IEC 61508. This
approach is regarded by the
regulator, at least in the UK, as appropriate to fulfill the ALARP
requirement (although of course
the courts are the final arbiters of that).

PBL

Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de



--
Les Chambers
les at chambers.com.au

https://www.chambers.com.au
https://www.systemsengineeringblog.com

+61 (0)412 648 992


_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230803/aa6ca04c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen-Shot-2019-11-06-at-4.10.29-PM-760x380.png
Type: image/png
Size: 342060 bytes
Desc: Screen-Shot-2019-11-06-at-4.10.29-PM-760x380.png
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230803/aa6ca04c/attachment-0001.png>