[SystemSafety] AI and the virtuous test Oracle - action now!

[Peter Bernard Ladkin]

Les,

On 2023-06-27 06:15 , Les Chambers wrote:
> ..... international bodies
> that currently regulate software-intensive Safety-Critical systems - who cling
> to regulating processes that have ceased to exist - are likely to be overrun
> and made redundant.

I don't see how this makes much sense. There are no international bodies that regulate 
software-intensive Safety-Critical systems (SISCS for short), except for IMO as far as I can tell. 
Except for IMO, regulation occurs at the level of nation-states, or the EU (whose member states have 
delegated certain regulatory activities to the EU in the sense that the EU writes directives that 
are then taken into national law by the members).

And as far as IMO goes, the level of SISCS in large ocean-going vessels seems to be of somewhat 
limited effect on the hazards of shipping (though I am open to reconsidering).

I don't know what "processes that have ceased to exist" you might be referring to; can you say?

Hazard and risk analysis (HRA) is regarded by IEC and ISO as key to standards involving safety 
considerations - that is explicitly what Guide 51 says - and Guide 51 says HRA shall be required in 
such standards, and tells us what it is. The regulation in many states of SISCS depends upon 
adherence to such standards. I don't see that the emergence of ML-based subsystems affects a 
requirement for HRA much at all - but I do see that traditional HRA is put in a quandary by how to 
evaluate systems with ML-based subsystems. The informal development standards applied by ML 
subsystem developers (often called "AI safety") don't work in traditional HRA assessments - rather, 
they do nominally work and rule ML-based subsystems out because reliability calculations are not 
possible.

However, I do see that there is considerably commercial pressure to approve safety-critical software 
which essentially uses ML-based subsystems for pervasive use, in particular in the road-vehicle 
sector, despite the lack of reliability assessment. But here there are, yes, regulatory hurdles. As 
well as considerable scepticism amongst many engineers. Not helped, of course, by revelations such 
as those by Handelsblatt, which suggests that Tesla knows of way more problems with its "Autopilot" 
SW than have been made public (Handelsblatt got hold of gigabytes of customer reports).

> In favour of organisations such as:
> 
> - The Center for Human-Compatible AI at UC Berkeley
> - The Future of Life Institute
> - The Center for AI Safety (CAIS)
> - Stanford Center for AI Safety

Can you name any reports on the reliability assessment of, say, control systems involving ML-based 
subsystems that any of those institutions have published? (There are quite a few such reports 
around, but those institutions are not where they come from.)

> .... This is a major
> inflection point in the evolution of intelligence. Carbon hosts will always be
> limited; silicon is unbounded.
Well, ChatGPT and its emergent behaviour certainly made the headlines recently. It's not new to me. 
I've been working on two projects since 2017 with language models based on word embedding (invented 
by Google ten years ago: Mikolov, Chen, Corrado and Dean). OpenAI and Google and Meta upped the 
scale and changed the application somewhat in 2021-2022, and then OpenAI puts a conversation bot on 
the open Internet and everybody goes bonkers. Because, rather than just a few devoted people (say, 
at the institutions you name) thinking about issues with chatbots, millions of people suddenly are.

It does seem worth emphasising that Chatbots based on word-embedding technology and control systems 
designed around ML-based environment-interpretation subsystems are two almost completely different 
technologies. What they have in common is ML technology.

The reason that word-embedding technology made what seems to be a quantum leap is the existence of 
huge corpora. You can train these things, if you wish, on more or less all the language that has 
ever been written down. And OpenAI (and maybe Google and Meta) did. Reported to have cost 
nine-figure sums of money. The CEO of OpenAI has said openly (and I believe him) that that is not a 
sustainable development model. Not necessarily for the cost, for there is lots of that kind of money 
in the world, but for the effort involved and the very special case of the entire environment being 
available (a universal corpus, as it were). Whereas the environment for road vehicle operation is 
not similarly available. It is also vastly more complex, as far as anyone can tell. We can't even 
say what it is. (Whereas conceptualising a corpus is something people have been able to do for 
millenia.) Apple and Google and who knows else have been training their road vehicle technology on 
public roads for well over the decade it took from the invention of word-embedding technology to the 
emergence of ChatGPT, and they are nowhere near "prime time" yet.

Further, I think you're wrong on the silicon level. There are hard physical limits to the 
development of current digital-computational processing units. Moore's Law cannot be open-ended. 
Many HW developers have pointed out we are reaching limits. I would be much more inclined to 
consider an "inflection point" when/if people get quantum computing to work. (I won't reiterate that 
well-rehearsed reasoning here.)

What does interest me is the political inflection point, if I may term it that. FLI put out its 
Slaughterbot video some years ago, and people such as Stuart Russell tried to get everyone to take 
it very seriously. We can thank our lucky stars that no capable national militaries seem to have 
taken it particularly seriously, for if they had we could well be in a world-wide political crisis 
in which no influential politician or national executive in any country could ever be seen in the 
open air ever again. Slaughterbot and similar threats have little to do with "intelligence", just 
with the capabilities of technology developed by people whom society has put in category of "AI 
research". But put a Chatbot on the Internet and all of a sudden the sky is falling.

PBL

Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de




-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230627/f7f28799/attachment.sig>