[SystemSafety] AI and the virtuous test Oracle - intelligence

[Prof. Dr. Peter Bernard Ladkin]

On 2023-06-30 20:34 , Steve Tockey wrote:
> I attended a conference earlier this year in Mysuru, India where an invited speaker, Dr. Seema Chopra from Boeing talked about adding in elements into AI systems whereby that system can be asked to “explain” why it gave the result that it did. Seemed like an interesting area of research that could maybe help in this area.

Interest in so-called "explainable AI" is around thirty years old. It can be thought to have started 
with the use of expert systems in medical-clinical diagnoses; physicians needed to have a rationale 
for the decisions they made aided by the systems. A short, though not necessarily completely 
accurate, history is at https://en.wikipedia.org/wiki/Explainable_artificial_intelligence

But for DLNNs and related technologies it's a wish, not a developed technology.

Interest in it has expanded because of the need in some circumstances to explain why we should trust 
the results of DLNNs and most other ML engines. We should be wary of assuming that because there is 
already a name for the task, that there is anything approaching a solution. (You can apply similar 
wariness to the term "AI" itself.)

It is possible to concoct rationales from the operation of expert systems or theorem provers - you 
look at how they used the rules to come up with their output. However, it is still really difficult 
to present those in human-surveyable form, as anyone who has used a theorem-prover in earnest will 
attest. Indeed, that is why there is a name for it and ongoing research. Note that the problem with 
trying similar for DLNNs is the complete lack of any kind of technical connection between the way 
the engine operates (its algorithm/s) and what would count as justification for a human assessor.

Note that there are significant difficulties, emanating from such phenomena as that noted by Ross 
Anderson and colleagues, that you can alter the outputs of a DLNN by permuting the training data. 
Intuitively, any justification for trusting DLNN output as a solution to whatever real-world problem 
you are addressing should be invariant under permutation of training data.

Note that when humans come up with concocted explanations for things (say, for things they did) it 
is called rationalisation and usually deprecated.

There are also domains in which it is hard to envisage how any "explainable AI" might work, for 
example, real-time control systems. For a pilot, for example, any such additional input is more 
likely to be a distraction than a help. If the airplane control system isn't doing what you want, 
then the immediate goal is to stop that behaviour and have it do what you want, not necessarily to 
have it explain itself.

-- 

PBL

Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de