[SystemSafety] Explainability of "Algorithms"

[Peter Bernard Ladkin]

The word "algorithm" has recently changed its meaning. It used to mean a von-Neumann-type procedural
method for attaining a specified goal, and the typical goals were mathematically or otherwise
rigorously specified. Now it seems to mean a programmed method for decision-making using statistical
machine-learning techniques typically on large datasets, where "programmed" means either typical
von-Neumann procedure (vN) or simulated-neural-network (SNN) behaviour. There is also the Valiant
bridging model (VBM) for parallel computation, which is demonstrably different from either.

I think it is probably right to include SNNs along with vNs as general computational procedures
whose specifics can be known as "algorithms". That would also be true of VBMs, but "no one" uses
VBMs. I think the reason "no one" uses VBMs is that computation has become hierarchical. The
assignment of subtasks amongst cores is seen as independent of what general algorithm is running on
the processor. The former is written by computer architects, and the latter still uses the vN serial
approach. The opposite has happened with SNNs. Their computation model is different, but the
substrate is a traditional vN serial-processor (which is then running on a multicore HW processor).

Today's modern students, though, are likely to pick up the notion of "algorithms" as meaning solely
sophisticated machine-learning devices. The traditional first courses in Algorithms and Data
Structures will need to add a "first lesson" which explains it is not going to be a course in how to
place advertisements on WWW pages.

Ed Felten, a computer-security expert at Princeton who was recently Deputy US Chief Technology
Officer in the Obama Administration, has published a blog post on (SNN-)algorithms and
"explainability"
https://freedom-to-tinker.com/2017/05/31/what-does-it-mean-to-ask-for-an-explainable-algorithm/  He
distinguishes some different ideas for what it might mean. I found it odd that he didn't address the
validation issue. It was brought up by Peter Leppick in comments (20170531@"1:41pm", presumably
EDT), to which Felten replies inter alia that

[begin quote]
But accuracy doesn’t necessarily require understanding. In some settings the error rate of an
algorithm can be estimated statistically with sufficient accuracy to have (enough) confidence in its
future accuracy. Or the algorithm can be designed to be self-monitoring, in the sense that it will
shut itself down and/or switch to a simpler algorithm if its error rate starts to creep up. In cases
like these, we might have confidence in its accuracy even without a detailed understanding of how it
works.
[end quote]

Are there really all that many cases in which the "error rate of an algorithm can be estimated
statistically with sufficient accuracy to have confidence in its future accuracy"? Can anybody here
say how that might go? (I know a little about renewal processes, for which you'd generally need
10-100 accumulated years of operational history for current confidence requirements, but
deep-learning SNN algorithms are almost the exact opposite of renewal processes, so almost nothing
would translate).

Say I give you a non-deterministic program which I tell you is based on a deep-learning SNN trained
on so-and-so-much data. Written by the world's best DL-SNN programmers (who swear they did it
"properly"). It is in your self-driving car and is meant to detect and avoid little kids who run out
into the road in front of it. It's been trained, and tested on 1,000,000 cases in which it has
worked as hoped (dare one say: as specified). What in the way of statistics can possibly tell you
with any degree of confidence that in the next ten cases (real cases, in your car) it won't actually
take aim at the kids and try to run them over? Surely you actually have to *know* some things about
the behaviour of the DL-SNN algorithm in order to draw any such conclusions? You can't draw
conclusions about behaviour out of thin air.

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20170628/492eb378/attachment.sig>