[SystemSafety] How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?

Mon Apr 18 16:44:35 CEST 2016

-       If you are conducting realistic road-tests, aren’t you putting the public *just* as much at risk as if you carried out a well-monitored progressive deployment?

I suspect that Google, Tesla and now Daimler with the E Class, have probably all conducted a lot of closed road testing (and prob. still do) before deployment and in the background and there was probably lots of “well monitored progressive development”. We just don’t see much of the detail, we just get the headlines.

- Why burn all the resources running around test cars?

Because the real world is noisy and pretty much unconstrained and there is only so much you can do and learn on a closed test track, the 64 billion dollar question being interacting with the environment. My best guess is that a large part of the Google program (and probably Tesla’s) is to collect vast amounts of data on what the real world looks like to their systems so they can rerun test drives as many times as necessary.

-       seems to me that there should be some licensing obligation to have a “black box” recorder that monitors the success of the driverless vehicles. Perhaps there already is.

At the moment I think the situation is “it depends”. In California the regulators have developed a set of requirements for AV’s, but in other states regulators have less onerous requirements for example in California regulators require reports of “incidents” but in Washington State they don’t. Google at least appears to have a complete set of data logging, Tesla has some logging, but their system is only “intended” for on- highway use so may be less all encompassing than Google’s. It is also reasonable to assume that like Goole Tesla also has an in-house test fleet. I have no idea what Daimler are up to.

-       The biggest challenge of any licensed / manageably-safe deployment is to encourage improvements that make the system improve in safety as rapidly as practical as experience develops. 

I think Tesla have made on-air updates to their system several times already, partly as a response to owners posting what they described as crazy behaviour on youtube e.g. reading newspapers while driving and reported issues e.g. inconsistent headway to lead vehicles. Tesla have also made at least one change after discussions with regulators in Singapore. Google have a much smaller fleet of vehicles so a much smaller  update problem and they only have that test fleet. Again no information on Daimler’s new E Class.

A more interesting question is what do people think that a satisfactory set of criteria for self driving cars would be? I’ve seen proposals for a driving test for such vehicles mirroring the driving test that new drivers have to pass which is reasonable a one level but perhaps not at others, e.g. Alvery’s comments on system improvement in as much as a new drivers skill increases as they do more driving.

As a first approximation I think it could include all or some of the following

-       Formal evaluation of the system design and architecture by regulators as per flight control systems

-       Skill evaluation testing, specific tests t(possibly large) to show skill in maintaining lane, distance to vehicle ahead, conformance to speed limits and so on.

-       Requirements for black box recording to evaluate responsibility when accidents occur.

-       Conformance fleet requirements for ongoing data collection and evaluation.

Discuss!

From: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of GRAZEBROOK, Alvery N
Sent: 18 April 2016 13:37
To: systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?

I’m not sure whether I’m missing the point here, but …

-       If you are conducting realistic road-tests, aren’t you putting the public *just* as much at risk as if you carried out a well-monitored progressive deployment? Why burn all the resources running around test cars? 

-       It seems to me that there should be some licensing obligation to have a “black box” recorder that monitors the success of the driverless vehicles. Perhaps there already is.

-       The biggest challenge of any licensed / manageably-safe deployment is to encourage improvements that make the system improve in safety as rapidly as practical as experience develops. 

Cheers,

            Alvery

** The opinions expressed here are my own, not necessarily those of my employer.

From: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of Matthew Squair
Sent: 18 April 2016 1:04 PM
To: martyn at thomas-associates.co.uk
Cc: systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?

Sure, making a bunch of assumptions. 

But I'd personally be comfortable after a couple of months of realistic road trials. 

Matthew Squair

MIEAust, CPEng

Mob: +61 488770655

Email; Mattsquair at gmail.com

Web: http://criticaluncertainties.com

On 18 Apr 2016, at 5:54 PM, Martyn Thomas <martyn at thomas-associates.co.uk> wrote:

On 18/04/2016 08:25, Peter Bishop wrote:

According to the Rand graph, we would need 1 million miles for a 95%

confidence that the driverless car crash rate is a least as good as a

driven one.

i.e. a fleet of 100 cars driving 10000 miles each for one year.

... under the same range and frequency of conditions as the comparable
fleet with human drivers.

Martyn

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE

This email and its attachments may contain confidential and/or privileged information.  If you have received them in error you must not use, copy or disclose their content to any person.  Please notify the sender immediately and then delete this email from your system.  This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail. 

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788.  Registered office:  Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20160418/00421a5f/attachment-0001.html>