[SystemSafety] AI and the virtuous test Oracle

Thu Jun 22 23:46:05 CEST 2023

RE your comment, “A neural network is software logic.”

A neural Network is a component of an AI agent. An AI agent is a suite of 
general purpose software tools. The agent is configured from a dataset.

ChatGPT goes on (see dialog below): “Synthesizing an AI agent involves training 
the machine learning model on a large dataset.” 

This indicates that the training dataset determines the agent’s behaviour 
(transfer function - inputs to outputs). In the case of a Tesla motor vehicle, 
the inputs are camera sensors and the outputs will soon be controls to the 
accelerator, brake and steering. 

RE your comment, “The problem is not AI. It is the exhaustive deterministic 
relationship between the undesired losses and the system states. Anything 
breaking this, breaks the current safety paradigm. And AI is not the single 
thing to achieve this.”

Could you please expand on this idea. Possibly using an example.

How would your thought apply here: I am a driver at the wheel of a Tesla 
travelling at 110 km/h on the freeway. I have switched the vehicle to full 
self-driving mode. I am fast asleep. The FSD agent detects a false positive – a 
lady with a pram. It jams on the brakes initiating a rear-end collision with a 
following semi. 
Q: In this case how is it that, “The problem is not AI”

Les

————— ChatGPT Dialog ——————

ChatGPT prompt:  
“Describe an AI agent from a technical perspective. What are its components and 
how is it synthesized?”

Response: 
An AI agent, from a technical perspective, typically consists of several 
components that work together to perform specific tasks. Here are the key 
components and how they are synthesized:

1. Data: The foundation of any AI agent is data. It can be in various forms 
such as text, images, audio, or structured data. This data is used for training 
and providing context to the AI model.

2. Machine Learning Model: The AI agent utilizes a machine learning model to 
process and generate responses. These models are trained on vast amounts of 
data and learn patterns to make predictions or generate outputs.

3. Natural Language Processing (NLP): NLP is a crucial component for 
understanding and processing human language. It involves tasks like language 
translation, sentiment analysis, intent recognition, and entity extraction. NLP 
enables the AI agent to comprehend user inputs and generate relevant responses.

4. Knowledge Representation: Knowledge representation refers to how information 
is stored and organized within the AI agent. It can employ various techniques 
such as semantic networks, ontologies, or graph databases to store facts and 
relationships. This allows the agent to access and utilize knowledge when 
responding to user queries.

5. Dialogue Management: Dialogue management focuses on handling the flow of 
conversation. It involves tracking the dialogue context, managing user requests 
and responses, and generating meaningful and coherent replies. Reinforcement 
learning techniques can be applied to improve dialogue management.

6. Integration and APIs: AI agents can integrate with external systems and 
services using APIs (Application Programming Interfaces). This allows them to 
access relevant data sources, databases, or external functionalities to provide 
more comprehensive and accurate responses.

7. User Interface: The AI agent needs a user interface to interact with users 
effectively. It can be a chat interface, voice-based interface, or a 
combination of both. The UI should provide a seamless experience for users to 
interact with the agent.

Synthesizing an AI agent involves training the machine learning model on a 
large dataset. This process includes steps like preprocessing the data, 
designing the model architecture, selecting appropriate algorithms, training 
the model using optimization techniques, and fine-tuning it for specific tasks. 
The trained model is then integrated with other components, such as NLP, 
knowledge representation, and dialogue management, to create a functioning AI 
agent. Regular updates, feedback loops, and continuous improvement are 
essential to enhance the agent's performance over time.

--------END Chat ----------

> C2 - Confidential
> 
> The problem is not AI. It is the exhaustive deterministic relationship 
between the undesired losses and the system states. Anything breaking this, 
breaks the current safety paradigm. And AI is not the single thing to achieve 
this.
> 
> AttentionÂ : due to increased cybersecurity screening I may receive external 
emails 30 minutes after their emission.
> RESTRICTED
> 
> Bertrand RICQUE
> Safety and security standardisation Expert
> Program Support Manager
> Optronics and Defence Division/Customer Support 
> Safran Electronics & Defense 
> 
> P +33 (0)1 58 11 96 82   M +33 (0)6 87 47 84 64 
> bertrand.ricque at safrangroup.com
> 102 Avenue de Paris
> 91300 MASSY FRANCE 
> www.safran-electronics-defense.com 
> 
> > -----Message d'origine-----
> > DeÂ : systemsafety <systemsafety-bounces at lists.techfak.uni-bielefeld.de>
> > De la part de Phil Koopman
> > EnvoyÃ©Â : jeudi 22 juin 2023 03:32
> > Ã€Â : les at chambers.com.au; systemsafety at lists.techfak.uni-bielefeld.de
> > ObjetÂ : Re: [SystemSafety] AI and the virtuous test Oracle
> > 
> > CAUTION:  This message originated from an outside organization. In case of
> > suspicion, click on "Report to SAFRAN Security" from the Outlook ribbon.
> > 
> > Les,
> > 
> > Since you welcome riffs, I have something that is not as all-encompassing,
> > but might have more immediate application.
> > 
> > I propose that to the degree that "AI" technology is deployed in a way that
> > supplants practical human judgement, the manufacturer of that system (in
> > some cases just the AI part if it is an add-on component) should be held
> > accountable for any action (or inaction) that, if associated with the human
> > that was supplanted, would have constituted negligence.Â  This should 
include
> > situations in which a human is put in an untenable situation of supervising 
an
> > AI in a way that puts unreasonable demands upon them, amounting to a
> > "moral crumple zone"
> > approach
> > (https://urldefense.com/v3/__https://papers.ssrn.com/sol3/papers.cfm?ab
> > stract_id=2757236__;!!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4
> > gg9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-DS0QOuOFaUivQZX7h$ ).
> > Liability/negligence if an AI is in substantive control of such a situation 
should
> > attach to the manufacturer.
> > 
> > This leads to a more narrow oracle, but perhaps still useful, than you
> > propose. If a loss event is caused by a lack of "reasonable" behavior by an 
AI,
> > the manufacturer is on the hook for negligence, and the AI/manufacturer
> > owes a duty of care the same as the human who was supplanted would have
> > owed to whoever/whatever might be affected by that negligence. It has the
> > advantage of reusing existing definitions of "reasonable person" that have
> > been hammered out over decades of law. (To be sure that is not in the form
> > of an engineering specification, but case law has a pretty robust set of
> > precedents, such as crashing into something after your properly functioning
> > vehicle ran a red light is likely to lead to the driver being found 
negligent.)
> > 
> > This does not require the AI to behave the same as people, and is not a 
full
> > recipe for "safe" AI. But it puts a floor on things in a way that is 
readily
> > actionable using existing legal mechanisms and theories. If a reasonable
> > person would have avoided a harm, any AI that fails to avoid the harm would
> > be negligent.
> > 
> > I've worked with a lawyer to propose this approach for automated vehicles,
> > and it is starting to get some traction. What I write in this post (above) 
is a
> > generalization of the concept beyond the narrow automated vehicle
> > application.
> > Details here:
> > https://urldefense.com/v3/__https://safeautonomy.blogspot.com/2023/05
> > /a-liability-approach-for-
> > automated.html__;!!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4gg
> > 9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-DS0QOuOFaUh295b5I$
> > 
> > -- Phil
> > 
> > 
> > On 6/21/2023 7:14 PM, Les Chambers wrote:
> > > Hi All
> > >
> > > I find myself reflecting on what will become of us.
> > > As systems engineering best practice is overrun by AI.
> > >
> > > Practitioners report that neural networks are eating code.
> > > Example 1: The vector field surrounding a Tesla motor vehicle is an
> > > output of a neural network, not the result of software logic. Soon the
> > > neural net - not code - will generate controls. The size of the code
> > > base is reducing.  (Elon
> > > Musk)
> > > Example 2: the ChatGPT transformer code base is only 2000 LOC (Mo
> > > Gawdat
> > > https://urldefense.com/v3/__https://youtu.be/bk-
> > nQ7HF6k4__;!!Dl6pPzL6!
> > >
> > dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4gg9Cz063kcikC8CIr0YMf2lF9o5xN
> > rnA0Av
> > > -DS0QOuOFaUpIawiVG$ )
> > >
> > > The intelligence resides in terabytes of data, perceptrons and
> > > millions of weighting parameters. All are gathered by automated means.
> > > Not subject to human review.
> > >
> > > Ergo what will become of our trusty barriers to dangerous failure:
> > > 1. Safety functions - gone
> > > 2. Verification - gone
> > > 3. Code reviews - gone
> > > 4. Validation - How?
> > >
> > > On validation, may I suggest the moral AI. A test oracle built on a
> > > virtuous dataset, capable of interrogating the target system to
> > > determine virtue. Test outcomes will morph from pass/failure to
> > moral/immoral.
> > >
> > > Credible industry players have predicted that soon we will have AIs
> > > orders of magnitude smarter than us. Especially when they start talking 
to
> > each other.
> > > The bandwidth will be eye-watering - the increase in intelligence, 
vertical.
> > >
> > > New barriers are required. Time to develop an AI that is on our side â€“
> > > the side of ethics and the moral life. An adult in the room if you
> > > like. We should birth this creature now and raise it as good parents.
> > >
> > > Let us not panic. May I put the proposition: virtue, like creativity,
> > > can be algorithmic.
> > > I have a sense of starting from the beginning - tabula rasa. I suggest
> > > that high-level thinking on the subject could begin with ChatGPT prompts:
> > > 1. What is the stoic philosopherâ€™s concept of virtue?
> > > 2. What are the elements of philosophy relevant to AI?
> > >
> > > Let us not forget our engineering mission: Guardians of the divine
> > > Logos, the organizing principle of the universe, responsible for its
> > > creation, maintenance, and order.
> > >
> > > Would anyone care to riff on this?
> > >
> > > Les
> > >
> > > --
> > >
> > > Les Chambers
> > >
> > > les at chambers.com.au
> > > systemsengineeringblog.com
> > >
> > > +61 (0)412 648 992
> > > _______________________________________________
> > > The System Safety Mailing List
> > > systemsafety at TechFak.Uni-Bielefeld.DE
> > > Manage your subscription:
> > > https://urldefense.com/v3/__https://lists.techfak.uni-bielefeld.de/mai
> > >
> > lman/listinfo/systemsafety__;!!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaC
> > GJCz
> > > dLlz4gg9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-DS0QOuOFaUjXMdlNF$
> > >
> > 
> > --
> > Prof. Phil Koopman   koopman at cmu.edu
> > (he/him)
> > https://urldefense.com/v3/__https://users.ece.cmu.edu/*koopman/__;fg!
> > !Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4gg9Cz063kcikC8CIr0YM
> > f2lF9o5xNrnA0Av-DS0QOuOFaUnsUrevc$
> > 
> > _______________________________________________
> > The System Safety Mailing List
> > systemsafety at TechFak.Uni-Bielefeld.DE
> > Manage your subscription:
> > https://urldefense.com/v3/__https://lists.techfak.uni-
> > bielefeld.de/mailman/listinfo/systemsafety__;!!Dl6pPzL6!dV6V79CEWJVLcd
> > XXS5n2wYWdaCGJCzdLlz4gg9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-
> > DS0QOuOFaUjXMdlNF$
> 
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des 
informations confidentielles, Ãªtre soumis aux rÃ¨glementations relatives au 
contrÃ´le des exportations ou ayant un caractÃ¨re privÃ©. S'ils ne vous sont 
pas destinÃ©s, nous vous signalons qu'il est strictement interdit de les 
divulguer, de les reproduire ou d'en utiliser de quelque maniÃ¨re que ce soit 
le contenu. Toute exportation ou rÃ©exportation non autorisÃ©e est interdite Si 
ce message vous a Ã©tÃ© transmis par erreur, merci d'en informer l'expÃ©diteur 
et de supprimer immÃ©diatement de votre systÃ¨me informatique ce courriel ainsi 
que tous les documents qui y sont attachÃ©s."
> ******
> " This e-mail and any attached documents may contain confidential or 
proprietary information and may be subject to export control laws and 
regulations. If you are not the intended recipient, you are notified that any 
dissemination, copying of this e-mail and any attachments thereto or use of 
their contents by any means whatsoever is strictly prohibited. Unauthorized 
export or re-export is prohibited. If you have received this e-mail in error, 
please advise the sender immediately and delete this e-mail and all attached 
documents from your computer system."
> #

--

Les Chambers

les at chambers.com.au

+61 (0)412 648 992