[SystemSafety] AI and the virtuous test Oracle - AI properties

Mon Jul 3 21:53:36 CEST 2023

Hi Michael
Good to hear that some structured thought is occurring on AI in the Safety-
Critical domain. When will ISO TR 5469 be available to the engineering 
community?
Is anyone publishing thoughts, elsewhere, for example, on “properties that need 
to be established.”

I have the sense that, in the light of AI, we're going to have to rethink 
systems engineering sharpish like.

It’s good advice: In order to discover new lands, one must be willing to lose 
sight of the shore for a very long time. - André Gide

Les

> A very interesting thread as well. Hello everybody!
> 
> I have worked with a group of people on an ISO TR 5469 regarding functional 
safety and AI. Some people who didn't read it but comment it uttered it would 
standardise AI. Actually the idea was different: there were AI specialists and 
safety specialists meeting to talk about the use of AI in FS applications and 
the first thing we found to be needed was that we search for a common language 
to speak to each other. As some know there are different concepts of "risk" in 
different disciplines and it is good to align first that the view of the groups 
is the same. Now the result is for me rather that this TR explains basic safety 
concepts to readers who want to fulfil safety standards with systems 
incorporating AI. It doesn't give a clear metric for expected integrity but it 
enables the communities to talk to each other and find solutions.
> 
> Why was this started? Because the AI community knows that the trust in what 
the systems do isn't there. So the logics is that if a standard would prescribe 
rules - maybe linked to certain technologies where evidence can be provided for 
certain properties that need to be established. This is why I must say that 
these catalogs of criteria are already heavily discussed. The biggest group 
involved into the creation of the TR at least in Germany were test houses. At 
the same time you also say that there are several resources that give rulesets 
and I also say I have seen six or seven different concepts. The fun part about 
that is that I rarely see the same concept in two of them, they are all using a 
different base. I also didn't find one ruleset that would give criteria for 
when the system really is working safe enough. One standard seems to be in 
favour at the moment that uses a catalog of criteria provided by one of the 
Fraunhofer institutes here in Germany as basis. So for me the time is right for 
the discussion.
> 
> About philosophy lessons. I fear future generations of engineers will use the 
future chatGPTs for their research, and if they are intelligent enough they 
will look for proof that what was provided is correct. I am sometimes sceptical 
where this leads. I had a discussion about numbers and figures days ago and 
told that I don't accept any diagnostic coverage value if it diagnoses only bus 
errors but not the problem - the drift of the sensor value. It was a little 
scary for me that I needed more than two minutes to convince that this is the 
right way. As logics also is a part of philosophy I'd start with that one. All 
the rest seems to be a very good idea but secondary (by the way I have read two 
books by technology philosopher Kornwachs). A similar problem seems to me that 
all teachers need to study their profession but a good background in education 
or didactics is underrepresented in the curriculum.
> 
> Besten GruÃŸ!
> Michael
> 
> --
> Michael KINDERMANN (he/him)
> Head of Functional Safety
> Team Leader Safety & Security
> Dpt. Global Compliance
> Phone: +49 621 776-2608
> 
> Pepperl+Fuchs SE, Mannheim
> 
> >
> 
> Pepperl+Fuchs SE, Mannheim
> VorstÃ¤nde/Board members: Dr.-Ing. Gunther Kegel (Vors.), Werner Guthier 
(stellv. Vors.), Mehmet Hatiboglu, Lutz Liebers, Reiner MÃ¼ller, Florian Ochs
> Vorsitzende des Aufsichtsrats/Chairwoman of the supervisory board: Monika 
MÃ¼ller-Michael
> Registergericht/Register Court: AG Mannheim HRB 737016 - UST-ID Nr. DE 
143877372
> 
> -----UrsprÃ¼ngliche Nachricht-----
> > Von: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-
> > bielefeld.de] Im Auftrag von Peter Bernard Ladkin
> > Gesendet: Dienstag, 27. Juni 2023 10:15
> > An: systemsafety at lists.techfak.uni-bielefeld.de
> > Betreff: Re: [SystemSafety] AI and the virtuous test Oracle - action now!
> >
> > ### EXTERNAL EMAIL ###
> >
> >
> > Les,
> >
> > On 2023-06-27 06:15 , Les Chambers wrote:
> > > ..... international bodies
> > > that currently regulate software-intensive Safety-Critical systems -
> > > who cling to regulating processes that have ceased to exist - are
> > > likely to be overrun and made redundant.
> >
> > I don't see how this makes much sense. There are no international bodies
> > that regulate software-intensive Safety-Critical systems (SISCS for short),
> > except for IMO as far as I can tell.
> > Except for IMO, regulation occurs at the level of nation-states, or the EU
> > (whose member states have delegated certain regulatory activities to the EU
> > in the sense that the EU writes directives that are then taken into 
national
> > law by the members).
> >
> > And as far as IMO goes, the level of SISCS in large ocean-going vessels 
seems
> > to be of somewhat limited effect on the hazards of shipping (though I am
> > open to reconsidering).
> >
> > I don't know what "processes that have ceased to exist" you might be
> > referring to; can you say?
> >
> > Hazard and risk analysis (HRA) is regarded by IEC and ISO as key to 
standards
> > involving safety considerations - that is explicitly what Guide 51 says - 
and
> > Guide 51 says HRA shall be required in such standards, and tells us what it 
is.
> > The regulation in many states of SISCS depends upon adherence to such
> > standards. I don't see that the emergence of ML-based subsystems affects a
> > requirement for HRA much at all - but I do see that traditional HRA is put 
in a
> > quandary by how to evaluate systems with ML-based subsystems. The
> > informal development standards applied by ML subsystem developers
> > (often called "AI safety") don't work in traditional HRA assessments - 
rather,
> > they do nominally work and rule ML-based subsystems out because
> > reliability calculations are not possible.
> >
> > However, I do see that there is considerably commercial pressure to approve
> > safety-critical software which essentially uses ML-based subsystems for
> > pervasive use, in particular in the road-vehicle sector, despite the lack 
of
> > reliability assessment. But here there are, yes, regulatory hurdles. As 
well as
> > considerable scepticism amongst many engineers. Not helped, of course, by
> > revelations such as those by Handelsblatt, which suggests that Tesla knows
> > of way more problems with its "Autopilot"
> > SW than have been made public (Handelsblatt got hold of gigabytes of
> > customer reports).
> >
> > > In favour of organisations such as:
> > >
> > > - The Center for Human-Compatible AI at UC Berkeley
> > > - The Future of Life Institute
> > > - The Center for AI Safety (CAIS)
> > > - Stanford Center for AI Safety
> >
> > Can you name any reports on the reliability assessment of, say, control
> > systems involving ML-based subsystems that any of those institutions have
> > published? (There are quite a few such reports around, but those 
institutions
> > are not where they come from.)
> >
> > > .... This is a major
> > > inflection point in the evolution of intelligence. Carbon hosts will
> > > always be limited; silicon is unbounded.
> > Well, ChatGPT and its emergent behaviour certainly made the headlines
> > recently. It's not new to me.
> > I've been working on two projects since 2017 with language models based on
> > word embedding (invented by Google ten years ago: Mikolov, Chen, Corrado
> > and Dean). OpenAI and Google and Meta upped the scale and changed the
> > application somewhat in 2021-2022, and then OpenAI puts a conversation bot
> > on the open Internet and everybody goes bonkers. Because, rather than just
> > a few devoted people (say, at the institutions you name) thinking about
> > issues with chatbots, millions of people suddenly are.
> >
> > It does seem worth emphasising that Chatbots based on word-embedding
> > technology and control systems designed around ML-based environment-
> > interpretation subsystems are two almost completely different technologies.
> > What they have in common is ML technology.
> >
> > The reason that word-embedding technology made what seems to be a
> > quantum leap is the existence of huge corpora. You can train these things, 
if
> > you wish, on more or less all the language that has ever been written down.
> > And OpenAI (and maybe Google and Meta) did. Reported to have cost nine-
> > figure sums of money. The CEO of OpenAI has said openly (and I believe him)
> > that that is not a sustainable development model. Not necessarily for the
> > cost, for there is lots of that kind of money in the world, but for the 
effort
> > involved and the very special case of the entire environment being 
available
> > (a universal corpus, as it were). Whereas the environment for road vehicle
> > operation is not similarly available. It is also vastly more complex, as 
far as
> > anyone can tell. We can't even say what it is. (Whereas conceptualising a
> > corpus is something people have been able to do for
> > millenia.) Apple and Google and who knows else have been training their
> > road vehicle technology on public roads for well over the decade it took 
from
> > the invention of word-embedding technology to the emergence of ChatGPT,
> > and they are nowhere near "prime time" yet.
> >
> > Further, I think you're wrong on the silicon level. There are hard physical
> > limits to the development of current digital-computational processing 
units.
> > Moore's Law cannot be open-ended.
> > Many HW developers have pointed out we are reaching limits. I would be
> > much more inclined to consider an "inflection point" when/if people get
> > quantum computing to work. (I won't reiterate that well-rehearsed
> > reasoning here.)
> >
> > What does interest me is the political inflection point, if I may term it 
that. FLI
> > put out its Slaughterbot video some years ago, and people such as Stuart
> > Russell tried to get everyone to take it very seriously. We can thank our 
lucky
> > stars that no capable national militaries seem to have taken it 
particularly
> > seriously, for if they had we could well be in a world-wide political 
crisis in
> > which no influential politician or national executive in any country could 
ever
> > be seen in the open air ever again. Slaughterbot and similar threats have 
little
> > to do with "intelligence", just with the capabilities of technology 
developed
> > by people whom society has put in category of "AI research". But put a
> > Chatbot on the Internet and all of a sudden the sky is falling.
> >
> > PBL
> >
> > Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
> > Tel+msg +49 (0)521 880 7319
> > Tel+http://atpscan.global.hornetsecurity.com/index.php?atp_str=dan4noM
> > CUfBIyfdve2Doy2zefN4Lu0TRKaS1U-XN27cRDB9W0CdYA-
> > rLa2izwfYaFuQYHLQxszB8hGsFUZecRJGFWwebrxzSJTQ0ilwKKy2Q0EffIKw139
> > HTdBdS1hqPhHNz8Pt4wOTAVEuWUjHlFmnsuI3eYdhLSTnedu7oPmhEbobtov
> > kq9ENIP40iO53uhfyw6ZZJqRCR4-LuCegp-
> > 8ZDntuWs0bMAA8ONpKIFpMLnR5qIKsWiXWkhzMzdlsAvLmm1gHZXz44DN-
> > ueYpQp6ZrH6MD-
> > 0HNY0nYt8kUSS0GFugtlxWO0t0PnfbWtPEGlY9bSYWy0i3_zbQjOjojR-
> > De2PidghgzsbRyIzo6Ix_JKl133GFDt4afXjqtIn0
> >
> >
> >
> 
> Wichtiger Hinweis:
> Diese E-Mail einschliesslich ihrer Anhaenge enthaelt vertrauliche und 
rechtlich geschuetzte Informationen, die nur fuer den Adressaten bestimmt sind. 
Sollten Sie nicht der bezeichnete Adressat sein, so teilen Sie dies bitte dem 
Absender umgehend mit und loeschen Sie diese Nachricht und ihre Anhaenge. Die 
unbefugte Weitergabe, das Anfertigen von Kopien und jede Veraenderung der E-
Mail ist untersagt. Der Absender haftet nicht fuer die Inhalte von veraenderten 
E-Mails.
> 
> Important Information:
> This e-mail message including its attachments contains confidential and 
legally protected information solely intended for the addressee. If you are not 
the intended addressee of this message, please contact the addresser 
immediately and delete this message including its attachments. The unauthorized 
dissemination, copying and change of this e-mail are strictly forbidden. The 
addresser shall not be liable for the content of such changed e-mails.
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-
bielefeld.de/mailman/listinfo/systemsafety

--

Les Chambers

les at chambers.com.au

+61 (0)412 648 992