[SystemSafety] AI Hallucination Cases

[M Ellims]

Peter wrote...

>> Exactly what negotiation goes on between car companies and government concerning
>> roadworthiness is beyond any knowledge of mine, but there is some.

There are several regulations (i.e. legally binding) that require specific work products be made available to the licensing bodies. Specifically the brake regulations UNECE Regulations 13 and 13H (brake functions) and steering (60 from memory). In regulation 13 this is Annex 18 (I have it open), the Annex's have different numbers in different regulations but the content is the same.

If you are familiar with the structure of ISO 26262 and its work products and read the Annex you can see the correspondence between them both. Thus it would appear that one has been designed to be compatible with the other - no idea which is the chicken and which is the egg.

BTW the egg came first; eggs were around long before chickens or dinosaurs.

Cheers.


-----Original Message-----
From: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of Prof. Dr. Peter Bernard Ladkin
Sent: 17 July 2025 08:36
To: les at chambers.com.au; systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] AI Hallucination Cases

Les,

On 2025-07-17 04:23 , Les Chambers wrote:
> Ergo, the proposition I believe this list should be debating is as follows:
> Given the inexorable replacement of procedural code by LLMs in safety critical
> systems, how can safety critical systems engineers hope to guarantee the
> general public protection from harm

This represents quite a disconnect with the world I know.

First,  in most of the safety-critical industry segments I know, safety-critical code (and coding) 
is governed by standards. There is DO 178C for avionics, ISO 26262 for road automotive vehicles, IEC 
61508 for industrial process plants (it is required by IEC 61511) and for much else except medical 
devices.

Second, none of the people I know working on Edition 3 of IEC 61508 see any way that a DLNN can 
fulfil the requirements of IEC 61508 Part 3. Many of them are feverishly working on a new ISO/IEC 
document, 22440, which will detail how encapsulated DLNNs may be used in critical systems which 
nominally fall under IEC 61508. A first attempt was made with ISO/IEC TR 5469. That was just a 
report, which I critiqued in https://scsc.uk/journal/index.php/scsj/article/view/32 . 22440 is 
*much* more detailed and attempts to be more rigorous about the development and use of DLNN modules 
in safety-critical systems.

Third, if you ask anybody whether procedural code "developed" by a LLM can satisfy the requirements 
of any of the above standards, the quick answer is "no". There are  over 50 different documentation 
requirements for code conformant to IEC 61508 Part 3. An LLM might hallucinate that  it has 
fulfilled them all but no wetware will take such a claim seriously.

Mutatis mutandis for DO 178C and ISO/IEC 26262.

Someone might want to say "there might be standards, but some industries just ignore them." Take 
your example, road vehicles. In countries with which I am familiar (namely, "first world"ish) road 
vehicles must be licensed. Licensing involves government approval. You can't just nail a few crates 
together in your garage, put wheels on and an engine in and go drive around public roads. You've got 
to licence it and the government won't do that unless someone has inspected your home-grown vehicle 
and determined it to be roadworthy. There are car companies, of course. But they nowadays adhere to 
standards such as ISO/IEC 26262 (in fact, they developed that standard themselves after determining 
that IEC 61508 could not be adapted to their domain). Exactly what negotiation goes on between car 
companies and government concerning roadworthiness is beyond any knowledge of mine, but there is 
some. Adherence to ISO/IEC 26262 for critical electronics including SW is part of it.

For commercial aerospace, it is even more transparent. The airworthiness criteria (certification 
regs) are freely available from USG (and other sites) and EASA. DO 178C is available to purchase 
from RTCA (equivalently, ED 12C from EUROCAE), as well as the accompanying guides on object-oriented 
development and use of formal methods.as well as the HW-SW interface. There are similar guides for 
ground-based software , ED 109A and ED 153.

These regulatory regimes governed by industrially-developed SW-development standards have been 
around for decades. They are not going away. None of them, as far as I know, make any provision for 
the use of procedural code generated by an LLM. I think your word "inexorable" is completely misplaced.

PBL

Prof. Dr. Peter Bernard Ladkin
Causalis Limited/Causalis IngenieurGmbH, Bielefeld, Germany
Tel: +49 (0)521 3 29 31 00

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety