[SystemSafety] Validating AI driven systems

Peter Bernard Ladkin ladkin at causalis.com
Tue Apr 18 18:23:44 CEST 2017



On 2017-04-18 16:54 , Martyn Thomas wrote:
> On 18/04/2017 08:02, Peter Bernard Ladkin wrote:
> 
>> Knight's article from the MIT Tech Review concerns a third aspect, namely when the networks are
>> trained/have trained themselves on masses of data ("deep learning"), how to explain
>> decisions/predictions the network then makes.
> 
> It may not be necessary to understand "why" the decision was made. It
> may be enough to come up with an acceptable story (true or false) after
> the event. 

There is a branch of philosophy called inference to the best explanation. I think it is important to
have a good explanation of the decision which is also a good explanation of why alternative
decisions weren't made. I agree with you that the exact mechanism by which the machine arrives at
the decision is secondary.

>> If you are talking about illness prediction, a large component of which has been known for a long
>> time to have a statistical character, then you do need to conjure up explanations, for you can't put
>> people on prophylactic therapy, which might itself be life-altering, without having a good idea why.
>> "The machine said to do it" is not a good reason.
> 
> "In my judgement, it was the action that had the highest probability of
> a good outcome" seems a good enough answer from doctors, so why not from
> a program?

It's not a good enough answer for me, much of the time. I usually want to know a bit more why. I
want the doctor, or a program, to infer to the best prognosis, and part of that is to explain the
inference.

>> If an adaptive control system does something funny, the question of how to rectify is non-trivial
> .....
> 
> If you were to have a potentially fatal (but actually not serious) bike
> accident that was your fault, would you "withdraw your licence to
> operate"? 

When I almost have a serious bike accident, my fault or not, then yes, I do think seriously about
whether I should quit riding my bike (for the rest of the day; for a couple days; for longer). Also,
there are days when I know my judgement is impaired when I do decide not to bike.

Waiting until it feels more up to the task doesn't seem to me to be a plausible option for SW.

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20170418/7e78ad2d/attachment.pgp>


More information about the systemsafety mailing list