[SystemSafety] Road to Damascus moment in Functional safety engineering - was FOSDEM talk by Paul Sherwood

[Derek M Jones]

Les,

> As far as I know there are no “currently prescribed” measures for evaluating
> the safety integrity of a large language model (LLM). At least nothing that,

The same can be said about humans.

> using the classical EN 50128esque criteria, would justify it’s deployment in a
> safety critical control system. The behaviour of this entity is a mystery even
> to it creators.

The same can be said about humans.

> My point is, “Do you really want this LLM component determining the trajectory
> of your motor vehicle when barrelling down a two lane road at a relative speed
> to the oncoming traffic of 200 km/h - when it’s developer doesn’t understand
> how it works, has not documented how it should work with a validatable
> specification and thinks it’s legitimate to foist it on you with a YOLO
> release under the cover of the name Full Self Driving?”

The same can be said about humans.

> [Phil & Rolf] They have no way to argue the predictive power of their safety
> case for real world safety outcomes other than experts say following
> prescribed engineering rigor requirements should be OK.

Sounds like how things currently work for non-LLM system.

The book "Killed by a Traffic Engineer" by Marshall is packed
with examples of traffic engineering rules that are treated
as having a scientific basis, but are not based on any research.
Traffic engineering, like software engineering, appears to be all
smoke and mirrors.

-- 
Derek M. Jones           Evidence-based software engineering
blog:https://shape-of-code.com