[SystemSafety] How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?

Sat Apr 23 10:12:47 CEST 2016

Apologies, for not being clear.

I wasn't referring to the learning component, but rather the security
problems of the Internet of Things. Cars whose software can be remotely
accessed, monitored and modified (as an example) being members of that set.

Given that a large proportion (hand waving here) of security exploits
utilise known or discovered bugs in the communications protocols that
support the IoT, you'd kind of think a logical response would be to
formally prove such protocols.

On Fri, Apr 22, 2016 at 9:10 PM, Mike Ellims <michael.ellims at tesco.net>
wrote:

> Hi Matthew,
>
>
>
> >  Really if ever there was a solid economic argument for deploying
> industrial scale formal method and proofs this would be it.
>
>
>
> To a machine learning system? How would you provide a formal proof that
> such a system had learnt the right response for all possible circumstances?
> I can conceive that it could be applied to the algorithms for learning but
> not to the learning itself. That is, you could show that the learning
> system does what it was specified to do, assuming that the specification is
> correct; but not that it was taught correctly or completely. For that I
> suspect that you will need some sort of statistical approach. How to do
> that is off course a major problem.
>
>
>
> And Hi Martyn
>
>
>
> > Recertification after software change.  Or do we just accept the huge
> attack surface that a fleet of AVs presents?
>
>
>
> For “recertification” Goggle’s approach to date seems to be to rerun all
> the driving done so far via simulation… I’m not sure what your implying
> with the comment on attack surfaces. Some far, as far as I can tell aside
> from updates there is not vehicle to vehicle communications. GPS is
> probably vulnerable to spoofing and jamming which could be an issue but one
> would hope that had been accounted for as it would count as a sensor
> failure…
>
>
>
> > The way in which AVs could change the safety of the total road transport
> system. Is anyone studying total accidents rather than AV accidents?
>
>
>
> Yes, lots and lots of people mostly government bodies that that collect
> the accident data in the first place and they tend to commission detailed
> studies from outside organization (that don’t quite answer the question
> your interested in). In addition to that there are a few
> manufacture/academic partnerships that study major road accidents in
> forensic detail alongside police (I know of one in Germany and one in the
> UK) which is intended to address many of the limitations to police
> investigations. In addition some of the big auto manufactures have their
> own departments e.g. VW have their own statistics department looking at
> this. In addition there is a large academic community concerned examining
> traffic accidents.
>
>
>
> As an aside, some time ago we were discussing wheels fall off of cars. I
> attempted to track down an answer to this from the online traffic stats as
> there is a field for it in the STATS19 form (filled out by police). However
> with some digging via email and a couple of phone calls to the Dept. of
> Transport it stopped dead with no answer because it’s a write-in field on
> the form and the data isn’t transferred to any of the computer systems. If
> it’s not on the computer they don’t want to know.
>
>
>
> Cheers.
>
>
>
> *From:* systemsafety [mailto:
> systemsafety-bounces at lists.techfak.uni-bielefeld.de] *On Behalf Of *Matthew
> Squair
> *Sent:* 22 April 2016 10:57
> *To:* Martyn Thomas
>
> *Cc:* Bielefield Safety List
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> I think that this is just another aspect of the Internet of Things
> challenge (software crisis #3).
>
>
>
> Really if ever there was a solid economic argument for deploying
> industrial scale formal method and proofs this would be it.
>
>
>
> Surely?
>
> Matthew Squair
>
>
>
> MIEAust, CPEng
>
> Mob: +61 488770655
>
> Email; Mattsquair at gmail.com
>
> Web: http://criticaluncertainties.com
>
>
> On 22 Apr 2016, at 2:20 AM, Martyn Thomas <martyn at 72f.org> wrote:
>
> Two issues.
>
>
>
> 1 Recertification after software change.  Or do we just accept the huge
> attack surface that a fleet of AVs presents?
>
>
>
> 2 The way in which AVs could change the safety of the total road transport
> system. Is anyone studying total accidents rather than AV accidents?
>
> Regards
>
>
>
> Martyn
>
>
> On 21 Apr 2016, at 17:47, Mike Ellims <michael.ellims at tesco.net> wrote:
>
> > This approach might be « safe ». I guess nobody has experience on this
> type of process.
>
>
>
> Mobileye has been around since 1999, Google have been letting cars drive
> themselves since 2009; I suspect they have probably got some experience by
> now. You would certainly hope so!
>
>
>
> > Whatever, it seems to have no intersection with the concept of
> satisfying safety requirements.
>
>
>
> That is possibly true at the top level for the complete system where some
> sort of statistical criteria may be more appropriate. However at the
> subsystem level I think that quite a number, or perhaps all of the
> principles laid out in IEC 16508 and ISO 26262 probably carry across quite
> well e.g. safety goals/requirements for system architecture  attributes
> such as fail silent/fail active,  warning and degradation concept etc. At
> lower levels requirements on the software for the inference engine design
> and code and requirements are applicable. For hardware  concepts such as
> safe failure fraction, failure detection percentage etc. would also be
> applicable.
>
>
>
> While having a dig around the interweb for information on Google’s self
> driving cars and the validation process I came across the following summary
> of drivers disengagements which gives a little insight into the process
> being used by Google and may be of interest and simulate further discussion.
>
>
>
>
> https://static.googleusercontent.com/media/www.google.com/en//selfdrivingcar/files/reports/report-annual-15.pdf
> <https://static.googleusercontent.com/media/www.google.com/en/selfdrivingcar/files/reports/report-annual-15.pdf>
>
>
>
>
>
> *From:* RICQUE Bertrand (SAGEM DEFENSE SECURITE) [
> mailto:bertrand.ricque at sagem.com <bertrand.ricque at sagem.com>]
> *Sent:* 21 April 2016 15:12
> *To:* Mike Ellims; 'Bielefield Safety List'
> *Cc:* systemsafety-bounces at lists.techfak.uni-bielefeld.de
> *Subject:* RE: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> This approach might be « safe ». I guess nobody has experience on this
> type of process.
>
>
>
> Whatever, it seems to have no intersection with the concept of satisfying
> safety requirements.
>
>
>
> Bertrand Ricque
>
> Program Manager
>
> Optronics and Defence Division
>
> Sights Program
>
> Mob : +33 6 87 47 84 64
>
> Tel : +33 1 58 11 96 82
>
> Bertrand.ricque at sagem.com
>
>
>
> *From:* Mike Ellims [mailto:michael.ellims at tesco.net
> <michael.ellims at tesco.net>]
> *Sent:* Thursday, April 21, 2016 3:35 PM
> *To:* RICQUE Bertrand (SAGEM DEFENSE SECURITE); 'Bielefield Safety List'
> *Cc:* systemsafety-bounces at lists.techfak.uni-bielefeld.de
> *Subject:* RE: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> Bertrand Ricque wrote
>
>
>
> > Safety critical software is not a question of time. It is a question of
> hunting bugs, in particular in uneasy access corners,
>
> > using dedicated methodologies, techniques and tools.
>
>
>
> That is true only up to a point, doing a bit of digging it seems that the
> majority of these systems are built on machine learning systems, so how you
> train them is going to be a large part of how “dependable” they are. Thus
> even if the code that implements the systems neural network  is perfect and
> is totally bug free (see below) the “dependability” of the final system is
>  on how good the training and testing  sets are which in turn is dependent
> on how many real world situations you can accumulate and present to the
> system.
>
>
>
> Hence Google’s approach of running around lots of cars to get as much
> information about road configurations, behaviour of other vehicles, issues
> (e.g. road signs obscured  by bushes) as possible which they can then
> combine with their humongous database of all the worlds roads.
>
>
>
> Tesla appears to uses a vision system from Mobileye, who’s website states
> on their planning systems;
>
>
>
> <snip> First, we apply supervised learning for predicting the near future
> based on the present. We require that the predictor will be
>
> differentiable with respect to the representation of the present. Second,
> we model a full trajectory of the agent using a
>
> recurrent neural network, where unexplained factors are modeled as
> (additive) input nodes. <snip>
>
>
>
>
>
>
>
> *From:* systemsafety [
> mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de
> <systemsafety-bounces at lists.techfak.uni-bielefeld.de>] *On Behalf Of *RICQUE
> Bertrand (SAGEM DEFENSE SECURITE)
> *Sent:* 21 April 2016 13:37
> *To:* Bielefield Safety List
> *Cc:* systemsafety-bounces at lists.techfak.uni-bielefeld.de
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> Safety critical software is not a question of time. It is a question of
> hunting bugs, in particular in uneasy access corners, using dedicated
> methodologies, techniques and tools.
>
>
>
> Say that you forgot to take into account in your software the fact that
> every 100 years bissextile years are not as every 4 years, you will never
> find it whatever the number of kilometres, cars and hours you use the
> system between 2001 and 2099…
>
>
>
> And whatever the good performance of your system during 99 years, there
> will be absolutely zero excuse for the consequent accidents …
>
>
>
> A good way to challenge the designers of such systems would be to make
> their children responsible for the damages …
>
>
>
> Bertrand Ricque
>
> Program Manager
>
> Optronics and Defence Division
>
> Sights Program
>
> Mob : +33 6 87 47 84 64
>
> Tel : +33 1 58 11 96 82
>
> Bertrand.ricque at sagem.com
>
>
>
> *From:* systemsafety [
> mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de
> <systemsafety-bounces at lists.techfak.uni-bielefeld.de>] *On Behalf Of *Nick
> Tudor
> *Sent:* Thursday, April 21, 2016 2:27 PM
> *To:* Matthew Squair
> *Cc:* Bielefield Safety List
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> This report has just come to my attention.  Stats based and an interesting
> read as it addresses most of the points made on this thread in one way or
> another:
>
>
>
> http://www.rand.org/pubs/research_reports/RR1478.html
>
>
> Nick Tudor
>
> Tudor Associates Ltd
>
> Mobile: +44(0)7412 074654
>
> www.tudorassoc.com
>
> <image001.jpg>
>
>
>
> *77 Barnards Green Road*
>
> *Malvern*
>
> *Worcestershire*
>
>
> *WR14 3LRCompany No. 07642673*
>
> *VAT No:116495996*
>
>
>
> *www.aeronautique-associates.com <http://www.aeronautique-associates.com>*
>
>
>
> On 18 April 2016 at 22:01, Matthew Squair <mattsquair at gmail.com> wrote:
>
> More that I don't see the value of multi million trip test programs that
> others might. ;)
>
> Matthew Squair
>
>
>
> MIEAust, CPEng
>
> Mob: +61 488770655
>
> Email; Mattsquair at gmail.com
>
> Web: http://criticaluncertainties.com
>
>
> On 18 Apr 2016, at 10:13 PM, Peter Bernard Ladkin <
> ladkin at rvs.uni-bielefeld.de> wrote:
>
>
>
> On 2016-04-18 14:03 , Matthew Squair wrote:
>
> But I'd personally be comfortable after a couple of months of realistic
> road trials.
>
>
> Hey, folks, we gotta volunteer!......... How you gonna line all those
> companies up, Matthew? :-)
>
> PBL
>
> Prof. Peter Bernard Ladkin, Faculty of Technology, University of
> Bielefeld, 33594 Bielefeld, Germany
> Je suis Charlie
> Tel+msg +49 (0)521 880 7319  www.rvs.uni-bielefeld.de
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>
>
>
> <image002.jpg>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> Virus-free. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> Virus-free. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <systemsafety at techfak.uni-bielefeld.de>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <systemsafety at techfak.uni-bielefeld.de>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>

-- 
*Matthew Squair*
BEng (Mech) MSysEng
MIEAust CPEng

Mob: +61 488770655
Email: MattSquair at gmail.com
Website: www.criticaluncertainties.com <http://criticaluncertainties.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20160423/6b244e22/attachment-0001.html>