[SystemSafety] Self-Driving Car Had a Fatal Accident ... (Norman, R 30 61 & many R 30.62)

[Peter Bernard Ladkin]

People have been arguing in Risks about the Tempe accident, and citing numbers (fatalities per
mile). Don Norman's post is at https://catless.ncl.ac.uk/Risks/30/61#subj1 and a number of people in
Risks 30.62 https://catless.ncl.ac.uk/Risks/30/62#subj13 are pointing out that he got his numbers
wrong. Don did issue a correction https://catless.ncl.ac.uk/Risks/30/62#subj14 But they are all just
playing with numbers also.

It is a little odd to see colleagues ignoring the difference between statistics and numerology.
Statistics involves drawing robust conclusions from data to a specified level of confidence.
Numerology is doing arithmetic on numbers.

One incident is not data, it is, at best, a datum. Consider: if the Risks discussion on the numbers
associated with self-driving cars had taken place on March 17th, it would have "concluded" that
self-driving cars were the safest form of road use ever known to mankind. On March 19th and since,
people are "concluding" that it is an order or two of magnitude "worse" than human-driven automotive
vehicles. The term "robust" is not applicable to inferences that flip so drastically between one day
and the next.

Not only that, but the technology of the accident vehicle may well change in response to the one
event, such that that event would not have occurred with the newer technology (operating as
designed). So those gathering numbers can start over again from 0. And this can go on and on.
Indeed, there is an argument (see below) that (a) this is how commercial aviation has addressed
digital automation, and (b) commercial aviation is by far the safest mode of travel by a number of
measures, and (c) continues to improve its safety record for high-performance jet transports using
any measure.

To show the weaknesses of lumping rare events by general category, let me consider three points of
view (please note that I am not advocating these points of view here: I am discussing them).

First point of view: there has been one fatal accident with self-driving cars.

There is one reasonably robust conclusion to be drawn from a sample of one, namely that the event
was not impossible. I think we knew that anyway.

Second point of view: there have been three fatal accidents with self-driving cars.

There was a fatal accident on May 7, 2016 near Williston in Florida, when a Tesla Model S on
"Autopilot" did not avoid a truck crossing the highway.

There was a second one on March 23, 2018 in Mountain View, California. Tesla has said that the
vehicle was travelling on "Autopilot" (see below).

Information about the March 23rd accident is preliminary, but it seems as if in both cases, as in
the Uber accident in Tempe, there was time for an alert human driver to register events and take
avoiding action. There is apparently little or no evidence that either of them did.

[The evidence: Tesla is saying that the March 23rd driver had about 5 seconds and 150m of
unobstructed view in which to intervene
https://www.nytimes.com/2018/03/31/business/tesla-crash-autopilot.html The NTSB said that both the
truck driver and the Tesla driver in the May 7th, 2016 collision had up to 10.4 seconds to register
the other vehicle, and the truck's line of sight was 1,132 ft.
https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf , section 2.2.2, p29.]

So, it does look as though, in both Tesla cases, the car was de facto driving itself.

Third view: there have been many fatal accidents with self-driving cars, many of them to cyclists
and even groups of cyclists.

This simple classification in fact applies to many road accidents. Drivers being distracted and not
paying attention to road conditions is an old theme. In this situation, the car is self-driving. I
remember a case of a woman searching for a music tape who hit a group of cyclists and killed a
number of them, I think in Colorado, but I can't seem to find it on the WWW (can someone help?).
Here is a recent one http://wishtv.com/2017/06/13/group-of-cyclists-mowed-down-by-driver-in-texas/ .
In these cases, there is no question that that the device being operated was not equipped to perform
the required tasks itself.

However, drawing conclusions using this classification is not going to tell us anything we didn't
know about automated road vehicles.

General Discussion.

For robust statistical conclusions, classification of events is crucial. There seems a clear
distinction between automobiles which require continuous human operation, and those which, under
certain road conditions, do not require continuous human operation, and those which (are intended
to) self-drive under all road conditions. The SAE has such a classification, with six categories:
see p24 of https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf .

The Tesla model counts as SAE Level 2
https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf . I believe (but have not
checked) that the Uber vehicle is SAE Level 5.

There is thus a prima facie reason to consider the first view to prevail, namely that the SAE
classification is widely used and we wish to draw conclusions about SAE Level 5 vehicles.

However, there are reasons why one might wish to draw conclusions about vehicles in multiple
categories, say Levels 2-5. For these, the second view is appropriate. Here are similarities.
(1) these vehicles were all operating in a "self-driving" mode (if the most recent report is correct)
(2) their minders all appear to have had time to detect and respond (way longer than the 1-2
seconds required for people "in the loop" to respond to an unanticipated event).
If nothing else, these similarities conform with observations about human supervisory control which
have been in the literature for decades.

In their report into the 2016 crash, the NTSB says "until automated vehicle systems mature, driver
engagement remains integral to the automated driving system" (op. cit. Section 2.1.1, p27). The
Board relates what Tesla's manual says of its "Autopilot" functions in Section 1.3.4, p13:

[begin quote]
.... with respect to the Autosteer system, the manual stated, “Warning: Autosteer is intended
for use only on highways and limited-access roads with a fully attentive driver” (p. 74). In
discussing restricted roads, the manual stated that “Autosteer is intended for use on freeways and
highways where access is limited by entry and exit ramps” (p. 75). The manual also stated that
“Autosteer is a hands-on feature. You must keep your hands on the steering wheel at all times” (p. 74).
[end quote]

The accident road in the 2016 crash did not have "access..limited by entry and exit ramps". Also,
"The NTSB concludes that the Tesla’s automated vehicle control system was not designed to, and did
not, identify the truck crossing the car’s path or recognize the impending crash....", (op. cit.,
Section 2.2.4, p30. This is also part of Finding 3, Section 3.1, p41).

In other words, a SAE Level 2 vehicle was being operated with automation in circumstances for which
the automation was not designed or validated, and encountered an obstruction  situation which it was
not designed to recognise, and indeed did not recognise. One might add that the car was travelling
(and had been programmed to travel) at 9mph over the posted limit. There are a lot of no-no's there.

Indeed, overspeed is known to be a factor in many car accidents over the decades. Even to be the
most common factor in fatal collisions. Is is banal, but nevertheless true, to say that, if all road
regulations were adhered to by all road users, fatal accidents and accidents with serious injury
would be, in comparison with today's figures, rare. Indeed, one main safety advantage proposed for
SAE Level 5 vehicles is that, unlike human operators, they will adhere to all road-use regulations.

As far as I know, the question is open (for the public) whether Uber's system is designed to detect
Elaine Herzberg pushing her bicycle across the road in Tempe outside of defined crossing points.
Which may be a different question from whether it is designed to detect slow-moving objects on
conflicting trajectories. The latter is a general question, to which the Herzberg event belongs. But
it may be that there are important characteristics of the Herzberg event which Uber's technology is
not designed to detect. For example, some reports suggest there may be "blind spots" in the car's
environmental sensing
https://www.reuters.com/article/us-uber-selfdriving-sensors-insight/ubers-use-of-fewer-safety-sensors-prompts-questions-after-arizona-crash-idUSKBN1H337Q
If there are such phenomena, we can presume that Uber will take action to mitigate them.

So a SAE Level 5 vehicle will thereby become a different-technology SAE Level 5 vehicle to which the
Herzberg event would not have happened. What happens then to any calculation of
fatality-per-miles-driven? Rather than say "it's still Level 5 so the numbers apply", it might be
more worth-while to subdivide categories: "Level 5 with sensing blind spots" versus "Level 5 without
sensing blind spots". The regulators might get there first and not licence any technology for
public-space trial which has any blind spots greater than such-and-such a size. That seems a more
operative approach than calculating numerical averages over constantly-changing categories.

Digitally-automated Commercial Aviation.

Compare with the situation with computer-controlled commercial aircraft. The first was the Airbus
A320, which suffered a fatal accident before it had been taken into service, on 1988-06-26. In its
first five years in service, it had suffered 4. (Actually, five years and three months spanned these
first four accidents, but the first happened technically before regular service introduction, on an
introductory pleasure flight.)

The A318/319/320/321 series has suffered 13 fatal accidents altogether, plus two deliberate hostile
(and successful) actions, from which no commercial aircraft is immune. So, from 4 accidents in 5
years to 13 accidents in nearly 30 years. From a few aircraft in service, to a few hundred, and now
a few thousand. From O(10 exp 4) fleet miles to O(10 exp 8) fleet miles. (Note: I imagine the fleet
miles today are around 100 million - they were around 55 million in 2007, as far as I remember.)

Certainly, some statistics can be formulated here, but the point I wish to emphasise is that the
most reliable and useful conclusions have been and are those drawn from detailed causal accounts of
all those accidents (and indeed the two hostile actions), not from statistics.

Consider a similar question to that which Norman and co. are attempting to answer. Is the A320
series "safe", or "safer than <something else>"? It seems to me that that is too broad a question to
be answered meaningfully as it stands. There is a lot of inappropriate crew handling spread across
those 15 fatal events. One can ask if the crew is somehow induced by the aircraft to indulge in such
inappropriate behaviour. That is a question which has been raised at every one of those 15
investigations, and answered variously. And a classic study, series of studies, by Woods and Sarter
identified the phenomenon of "mode confusion", which seems to have been all but eliminated nowadays
through combinations of technical change, operating-procedural changes and training changes.

The history validates this approach, which is to find out individually what might have been wrong,
and devise and propagate mitigation. The A320 aircraft of today are different beasts in many subtle
and not-so-subtle ways from those of 30 years ago. Statistical inference doesn't work very well
across changing technology. And mitigation may not always address a causal factor directly. The
response to a technological causal factor may well be altered crew behaviour (for example,
concerning high-altitude pitot icing); the response to a human factor might well be technological
(for example, a change to an interface to alleviate potential "mode confusion").

PBL

Peter Bernard Ladkin, Bielefeld, Germany
www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20180401/56668029/attachment.sig>