[SystemSafety] AI and the virtuous test Oracle - action now!

[Kindermann Michael]

A very interesting thread as well. Hello everybody!

I have worked with a group of people on an ISO TR 5469 regarding functional safety and AI. Some people who didn't read it but comment it uttered it would standardise AI. Actually the idea was different: there were AI specialists and safety specialists meeting to talk about the use of AI in FS applications and the first thing we found to be needed was that we search for a common language to speak to each other. As some know there are different concepts of "risk" in different disciplines and it is good to align first that the view of the groups is the same. Now the result is for me rather that this TR explains basic safety concepts to readers who want to fulfil safety standards with systems incorporating AI. It doesn't give a clear metric for expected integrity but it enables the communities to talk to each other and find solutions.

Why was this started? Because the AI community knows that the trust in what the systems do isn't there. So the logics is that if a standard would prescribe rules - maybe linked to certain technologies where evidence can be provided for certain properties that need to be established. This is why I must say that these catalogs of criteria are already heavily discussed. The biggest group involved into the creation of the TR at least in Germany were test houses. At the same time you also say that there are several resources that give rulesets and I also say I have seen six or seven different concepts. The fun part about that is that I rarely see the same concept in two of them, they are all using a different base. I also didn't find one ruleset that would give criteria for when the system really is working safe enough. One standard seems to be in favour at the moment that uses a catalog of criteria provided by one of the Fraunhofer institutes here in Germany as basis. So for me the time is right for the discussion.

About philosophy lessons. I fear future generations of engineers will use the future chatGPTs for their research, and if they are intelligent enough they will look for proof that what was provided is correct. I am sometimes sceptical where this leads. I had a discussion about numbers and figures days ago and told that I don't accept any diagnostic coverage value if it diagnoses only bus errors but not the problem - the drift of the sensor value. It was a little scary for me that I needed more than two minutes to convince that this is the right way. As logics also is a part of philosophy I'd start with that one. All the rest seems to be a very good idea but secondary (by the way I have read two books by technology philosopher Kornwachs). A similar problem seems to me that all teachers need to study their profession but a good background in education or didactics is underrepresented in the curriculum.

Besten Gruß!
Michael

--
Michael KINDERMANN (he/him)
Head of Functional Safety
Team Leader Safety & Security
Dpt. Global Compliance
Phone: +49 621 776-2608

Pepperl+Fuchs SE, Mannheim


>

Pepperl+Fuchs SE, Mannheim
Vorstände/Board members: Dr.-Ing. Gunther Kegel (Vors.), Werner Guthier (stellv. Vors.), Mehmet Hatiboglu, Lutz Liebers, Reiner Müller, Florian Ochs
Vorsitzende des Aufsichtsrats/Chairwoman of the supervisory board: Monika Müller-Michael
Registergericht/Register Court: AG Mannheim HRB 737016 - UST-ID Nr. DE 143877372

-----Ursprüngliche Nachricht-----
> Von: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-
> bielefeld.de] Im Auftrag von Peter Bernard Ladkin
> Gesendet: Dienstag, 27. Juni 2023 10:15
> An: systemsafety at lists.techfak.uni-bielefeld.de
> Betreff: Re: [SystemSafety] AI and the virtuous test Oracle - action now!
>
> ### EXTERNAL EMAIL ###
>
>
> Les,
>
> On 2023-06-27 06:15 , Les Chambers wrote:
> > ..... international bodies
> > that currently regulate software-intensive Safety-Critical systems -
> > who cling to regulating processes that have ceased to exist - are
> > likely to be overrun and made redundant.
>
> I don't see how this makes much sense. There are no international bodies
> that regulate software-intensive Safety-Critical systems (SISCS for short),
> except for IMO as far as I can tell.
> Except for IMO, regulation occurs at the level of nation-states, or the EU
> (whose member states have delegated certain regulatory activities to the EU
> in the sense that the EU writes directives that are then taken into national
> law by the members).
>
> And as far as IMO goes, the level of SISCS in large ocean-going vessels seems
> to be of somewhat limited effect on the hazards of shipping (though I am
> open to reconsidering).
>
> I don't know what "processes that have ceased to exist" you might be
> referring to; can you say?
>
> Hazard and risk analysis (HRA) is regarded by IEC and ISO as key to standards
> involving safety considerations - that is explicitly what Guide 51 says - and
> Guide 51 says HRA shall be required in such standards, and tells us what it is.
> The regulation in many states of SISCS depends upon adherence to such
> standards. I don't see that the emergence of ML-based subsystems affects a
> requirement for HRA much at all - but I do see that traditional HRA is put in a
> quandary by how to evaluate systems with ML-based subsystems. The
> informal development standards applied by ML subsystem developers
> (often called "AI safety") don't work in traditional HRA assessments - rather,
> they do nominally work and rule ML-based subsystems out because
> reliability calculations are not possible.
>
> However, I do see that there is considerably commercial pressure to approve
> safety-critical software which essentially uses ML-based subsystems for
> pervasive use, in particular in the road-vehicle sector, despite the lack of
> reliability assessment. But here there are, yes, regulatory hurdles. As well as
> considerable scepticism amongst many engineers. Not helped, of course, by
> revelations such as those by Handelsblatt, which suggests that Tesla knows
> of way more problems with its "Autopilot"
> SW than have been made public (Handelsblatt got hold of gigabytes of
> customer reports).
>
> > In favour of organisations such as:
> >
> > - The Center for Human-Compatible AI at UC Berkeley
> > - The Future of Life Institute
> > - The Center for AI Safety (CAIS)
> > - Stanford Center for AI Safety
>
> Can you name any reports on the reliability assessment of, say, control
> systems involving ML-based subsystems that any of those institutions have
> published? (There are quite a few such reports around, but those institutions
> are not where they come from.)
>
> > .... This is a major
> > inflection point in the evolution of intelligence. Carbon hosts will
> > always be limited; silicon is unbounded.
> Well, ChatGPT and its emergent behaviour certainly made the headlines
> recently. It's not new to me.
> I've been working on two projects since 2017 with language models based on
> word embedding (invented by Google ten years ago: Mikolov, Chen, Corrado
> and Dean). OpenAI and Google and Meta upped the scale and changed the
> application somewhat in 2021-2022, and then OpenAI puts a conversation bot
> on the open Internet and everybody goes bonkers. Because, rather than just
> a few devoted people (say, at the institutions you name) thinking about
> issues with chatbots, millions of people suddenly are.
>
> It does seem worth emphasising that Chatbots based on word-embedding
> technology and control systems designed around ML-based environment-
> interpretation subsystems are two almost completely different technologies.
> What they have in common is ML technology.
>
> The reason that word-embedding technology made what seems to be a
> quantum leap is the existence of huge corpora. You can train these things, if
> you wish, on more or less all the language that has ever been written down.
> And OpenAI (and maybe Google and Meta) did. Reported to have cost nine-
> figure sums of money. The CEO of OpenAI has said openly (and I believe him)
> that that is not a sustainable development model. Not necessarily for the
> cost, for there is lots of that kind of money in the world, but for the effort
> involved and the very special case of the entire environment being available
> (a universal corpus, as it were). Whereas the environment for road vehicle
> operation is not similarly available. It is also vastly more complex, as far as
> anyone can tell. We can't even say what it is. (Whereas conceptualising a
> corpus is something people have been able to do for
> millenia.) Apple and Google and who knows else have been training their
> road vehicle technology on public roads for well over the decade it took from
> the invention of word-embedding technology to the emergence of ChatGPT,
> and they are nowhere near "prime time" yet.
>
> Further, I think you're wrong on the silicon level. There are hard physical
> limits to the development of current digital-computational processing units.
> Moore's Law cannot be open-ended.
> Many HW developers have pointed out we are reaching limits. I would be
> much more inclined to consider an "inflection point" when/if people get
> quantum computing to work. (I won't reiterate that well-rehearsed
> reasoning here.)
>
> What does interest me is the political inflection point, if I may term it that. FLI
> put out its Slaughterbot video some years ago, and people such as Stuart
> Russell tried to get everyone to take it very seriously. We can thank our lucky
> stars that no capable national militaries seem to have taken it particularly
> seriously, for if they had we could well be in a world-wide political crisis in
> which no influential politician or national executive in any country could ever
> be seen in the open air ever again. Slaughterbot and similar threats have little
> to do with "intelligence", just with the capabilities of technology developed
> by people whom society has put in category of "AI research". But put a
> Chatbot on the Internet and all of a sudden the sky is falling.
>
> PBL
>
> Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
> Tel+msg +49 (0)521 880 7319
> Tel+http://atpscan.global.hornetsecurity.com/index.php?atp_str=dan4noM
> CUfBIyfdve2Doy2zefN4Lu0TRKaS1U-XN27cRDB9W0CdYA-
> rLa2izwfYaFuQYHLQxszB8hGsFUZecRJGFWwebrxzSJTQ0ilwKKy2Q0EffIKw139
> HTdBdS1hqPhHNz8Pt4wOTAVEuWUjHlFmnsuI3eYdhLSTnedu7oPmhEbobtov
> kq9ENIP40iO53uhfyw6ZZJqRCR4-LuCegp-
> 8ZDntuWs0bMAA8ONpKIFpMLnR5qIKsWiXWkhzMzdlsAvLmm1gHZXz44DN-
> ueYpQp6ZrH6MD-
> 0HNY0nYt8kUSS0GFugtlxWO0t0PnfbWtPEGlY9bSYWy0i3_zbQjOjojR-
> De2PidghgzsbRyIzo6Ix_JKl133GFDt4afXjqtIn0
>
>
>


Wichtiger Hinweis:
Diese E-Mail einschliesslich ihrer Anhaenge enthaelt vertrauliche und rechtlich geschuetzte Informationen, die nur fuer den Adressaten bestimmt sind. Sollten Sie nicht der bezeichnete Adressat sein, so teilen Sie dies bitte dem Absender umgehend mit und loeschen Sie diese Nachricht und ihre Anhaenge. Die unbefugte Weitergabe, das Anfertigen von Kopien und jede Veraenderung der E-Mail ist untersagt. Der Absender haftet nicht fuer die Inhalte von veraenderten E-Mails.

Important Information:
This e-mail message including its attachments contains confidential and legally protected information solely intended for the addressee. If you are not the intended addressee of this message, please contact the addresser immediately and delete this message including its attachments. The unauthorized dissemination, copying and change of this e-mail are strictly forbidden. The addresser shall not be liable for the content of such changed e-mails.