[SystemSafety] Explainability of "Algorithms"

[Peter Bernard Ladkin]

On 2017-06-28 07:44 , Peter Bernard Ladkin wrote:
> Say I give you a non-deterministic program which I tell you is based on a deep-learning SNN trained
> on so-and-so-much data. Written by the world's best DL-SNN programmers (who swear they did it
> "properly"). It is in your self-driving car and is meant to detect and avoid little kids who run out
> into the road in front of it. It's been trained, and tested on 1,000,000 cases in which it has
> worked as hoped (dare one say: as specified). What in the way of statistics can possibly tell you
> with any degree of confidence that in the next ten cases (real cases, in your car) it won't actually
> take aim at the kids and try to run them over? Surely you actually have to *know* some things about
> the behaviour of the DL-SNN algorithm in order to draw any such conclusions? You can't draw
> conclusions about behaviour out of thin air.
I maybe should have mentioned that this is a version of Nelson Goodman's "New Riddle of Induction",
otherwise known as the grue paradox.

Goodman pointed out that any evidence (before, say July 30, 2017) for the assertion "X is green" is
equally evidence for "X is grue", where "grue" is defined as "green before midnight on July 30,
2017, and blue thereafter".

Similarly, any behavioural-observational evidence for "this algorithm avoids little kids" before you
take possession of your self-driving car is equally evidence for "this algorithm avoids little kids
until the owner takes possession of the car, whereupon it aims at them".

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20170628/d100fba8/attachment.sig>