[SystemSafety] How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?

Sat Apr 23 15:28:21 CEST 2016

The LANGSEC community has done a lot of work in this area, they're
suggesting we need to refactor Postel's principle at a minimum.

On Sat, Apr 23, 2016 at 9:14 PM, Mike Ellims <michael.ellims at tesco.net>
wrote:

> The approach taken to date is to assume that the medium is unsafe i.e. you
> can get various attacks such as man in the middle etc. so that validation
> occurs at either end without any strong dependence on the transmission
> medium.
>
>
>
> The problem with this approach is that it has been demonstrated car
> manufactures are not always that good at internal hygiene, for example.
>
>
>
> The Bluetooth interface is not adequately separated from the vehicles
> internal network so you have access to the CAN bus externally.
>
> Not even very good at following their own or industry software standards
> such as not allowing software or calibration  updates while the vehicle is
> moving.
>
>
>
> Poor use of security, in general items that are considered to be safety
> related such as engines and transmission control systems are protected,
> sometimes quite strongly. However they tend to communicate via channels
> that are weakly protected e.g. CAN which means that if an item with low
> safety related level is subverted (such as the infotainment system) then if
> the vehicle architecture is poor then that can allow other systems to be
> spoofed.
>
>
>
> So to answer your question, in an increasing linked set of cars the threat
> is feasible if not yet exploited but it may well apply to most vehicles not
> just AV’s.
>
>
>
> This site http://www.autosec.org/publications.html
>
>
>
> Has a number of papers in this area such as Comprehensive Experimental
> Analyses of Automotive Attack Surfaces. A new paper I haven’t read “Fast
> and Vulnerable: A Story of Telematic Failures” on an initial scan looks
> interesting (I can’t find my read glasses… old , stupid and now blind).
>
>
>
> Cheers.
>
>
>
> *From:* Matthew Squair [mailto:mattsquair at gmail.com]
> *Sent:* 23 April 2016 09:13
> *To:* Mike Ellims
> *Cc:* Martyn Thomas; Bielefield Safety List
>
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> Apologies, for not being clear.
>
>
>
> I wasn't referring to the learning component, but rather the security
> problems of the Internet of Things. Cars whose software can be remotely
> accessed, monitored and modified (as an example) being members of that set.
>
>
>
> Given that a large proportion (hand waving here) of security exploits
> utilise known or discovered bugs in the communications protocols that
> support the IoT, you'd kind of think a logical response would be to
> formally prove such protocols.
>
>
>
>
>
>
>
> On Fri, Apr 22, 2016 at 9:10 PM, Mike Ellims <michael.ellims at tesco.net>
> wrote:
>
> Hi Matthew,
>
>
>
> >  Really if ever there was a solid economic argument for deploying
> industrial scale formal method and proofs this would be it.
>
>
>
> To a machine learning system? How would you provide a formal proof that
> such a system had learnt the right response for all possible circumstances?
> I can conceive that it could be applied to the algorithms for learning but
> not to the learning itself. That is, you could show that the learning
> system does what it was specified to do, assuming that the specification is
> correct; but not that it was taught correctly or completely. For that I
> suspect that you will need some sort of statistical approach. How to do
> that is off course a major problem.
>
>
>
> And Hi Martyn
>
>
>
> > Recertification after software change.  Or do we just accept the huge
> attack surface that a fleet of AVs presents?
>
>
>
> For “recertification” Goggle’s approach to date seems to be to rerun all
> the driving done so far via simulation… I’m not sure what your implying
> with the comment on attack surfaces. Some far, as far as I can tell aside
> from updates there is not vehicle to vehicle communications. GPS is
> probably vulnerable to spoofing and jamming which could be an issue but one
> would hope that had been accounted for as it would count as a sensor
> failure…
>
>
>
> > The way in which AVs could change the safety of the total road transport
> system. Is anyone studying total accidents rather than AV accidents?
>
>
>
> Yes, lots and lots of people mostly government bodies that that collect
> the accident data in the first place and they tend to commission detailed
> studies from outside organization (that don’t quite answer the question
> your interested in). In addition to that there are a few
> manufacture/academic partnerships that study major road accidents in
> forensic detail alongside police (I know of one in Germany and one in the
> UK) which is intended to address many of the limitations to police
> investigations. In addition some of the big auto manufactures have their
> own departments e.g. VW have their own statistics department looking at
> this. In addition there is a large academic community concerned examining
> traffic accidents.
>
>
>
> As an aside, some time ago we were discussing wheels fall off of cars. I
> attempted to track down an answer to this from the online traffic stats as
> there is a field for it in the STATS19 form (filled out by police). However
> with some digging via email and a couple of phone calls to the Dept. of
> Transport it stopped dead with no answer because it’s a write-in field on
> the form and the data isn’t transferred to any of the computer systems. If
> it’s not on the computer they don’t want to know.
>
>
>
> Cheers.
>
>
>
> *From:* systemsafety [mailto:
> systemsafety-bounces at lists.techfak.uni-bielefeld.de] *On Behalf Of *Matthew
> Squair
> *Sent:* 22 April 2016 10:57
> *To:* Martyn Thomas
>
>
> *Cc:* Bielefield Safety List
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> I think that this is just another aspect of the Internet of Things
> challenge (software crisis #3).
>
>
>
> Really if ever there was a solid economic argument for deploying
> industrial scale formal method and proofs this would be it.
>
>
>
> Surely?
>
> Matthew Squair
>
>
>
> MIEAust, CPEng
>
> Mob: +61 488770655
>
> Email; Mattsquair at gmail.com
>
> Web: http://criticaluncertainties.com
>
>
> On 22 Apr 2016, at 2:20 AM, Martyn Thomas <martyn at 72f.org> wrote:
>
> Two issues.
>
>
>
> 1 Recertification after software change.  Or do we just accept the huge
> attack surface that a fleet of AVs presents?
>
>
>
> 2 The way in which AVs could change the safety of the total road transport
> system. Is anyone studying total accidents rather than AV accidents?
>
> Regards
>
>
>
> Martyn
>
>
> On 21 Apr 2016, at 17:47, Mike Ellims <michael.ellims at tesco.net> wrote:
>
> > This approach might be « safe ». I guess nobody has experience on this
> type of process.
>
>
>
> Mobileye has been around since 1999, Google have been letting cars drive
> themselves since 2009; I suspect they have probably got some experience by
> now. You would certainly hope so!
>
>
>
> > Whatever, it seems to have no intersection with the concept of
> satisfying safety requirements.
>
>
>
> That is possibly true at the top level for the complete system where some
> sort of statistical criteria may be more appropriate. However at the
> subsystem level I think that quite a number, or perhaps all of the
> principles laid out in IEC 16508 and ISO 26262 probably carry across quite
> well e.g. safety goals/requirements for system architecture  attributes
> such as fail silent/fail active,  warning and degradation concept etc. At
> lower levels requirements on the software for the inference engine design
> and code and requirements are applicable. For hardware  concepts such as
> safe failure fraction, failure detection percentage etc. would also be
> applicable.
>
>
>
> While having a dig around the interweb for information on Google’s self
> driving cars and the validation process I came across the following summary
> of drivers disengagements which gives a little insight into the process
> being used by Google and may be of interest and simulate further discussion.
>
>
>
>
> https://static.googleusercontent.com/media/www.google.com/en//selfdrivingcar/files/reports/report-annual-15.pdf
> <https://static.googleusercontent.com/media/www.google.com/en/selfdrivingcar/files/reports/report-annual-15.pdf>
>
>
>
>
>
> *From:* RICQUE Bertrand (SAGEM DEFENSE SECURITE) [
> mailto:bertrand.ricque at sagem.com <bertrand.ricque at sagem.com>]
> *Sent:* 21 April 2016 15:12
> *To:* Mike Ellims; 'Bielefield Safety List'
> *Cc:* systemsafety-bounces at lists.techfak.uni-bielefeld.de
> *Subject:* RE: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> This approach might be « safe ». I guess nobody has experience on this
> type of process.
>
>
>
> Whatever, it seems to have no intersection with the concept of satisfying
> safety requirements.
>
>
>
> Bertrand Ricque
>
> Program Manager
>
> Optronics and Defence Division
>
> Sights Program
>
> Mob : +33 6 87 47 84 64
>
> Tel : +33 1 58 11 96 82
>
> Bertrand.ricque at sagem.com
>
>
>
> *From:* Mike Ellims [mailto:michael.ellims at tesco.net
> <michael.ellims at tesco.net>]
> *Sent:* Thursday, April 21, 2016 3:35 PM
> *To:* RICQUE Bertrand (SAGEM DEFENSE SECURITE); 'Bielefield Safety List'
> *Cc:* systemsafety-bounces at lists.techfak.uni-bielefeld.de
> *Subject:* RE: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> Bertrand Ricque wrote
>
>
>
> > Safety critical software is not a question of time. It is a question of
> hunting bugs, in particular in uneasy access corners,
>
> > using dedicated methodologies, techniques and tools.
>
>
>
> That is true only up to a point, doing a bit of digging it seems that the
> majority of these systems are built on machine learning systems, so how you
> train them is going to be a large part of how “dependable” they are. Thus
> even if the code that implements the systems neural network  is perfect and
> is totally bug free (see below) the “dependability” of the final system is
>  on how good the training and testing  sets are which in turn is dependent
> on how many real world situations you can accumulate and present to the
> system.
>
>
>
> Hence Google’s approach of running around lots of cars to get as much
> information about road configurations, behaviour of other vehicles, issues
> (e.g. road signs obscured  by bushes) as possible which they can then
> combine with their humongous database of all the worlds roads.
>
>
>
> Tesla appears to uses a vision system from Mobileye, who’s website states
> on their planning systems;
>
>
>
> <snip> First, we apply supervised learning for predicting the near future
> based on the present. We require that the predictor will be
>
> differentiable with respect to the representation of the present. Second,
> we model a full trajectory of the agent using a
>
> recurrent neural network, where unexplained factors are modeled as
> (additive) input nodes. <snip>
>
>
>
>
>
>
>
> *From:* systemsafety [
> mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de
> <systemsafety-bounces at lists.techfak.uni-bielefeld.de>] *On Behalf Of *RICQUE
> Bertrand (SAGEM DEFENSE SECURITE)
> *Sent:* 21 April 2016 13:37
> *To:* Bielefield Safety List
> *Cc:* systemsafety-bounces at lists.techfak.uni-bielefeld.de
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> Safety critical software is not a question of time. It is a question of
> hunting bugs, in particular in uneasy access corners, using dedicated
> methodologies, techniques and tools.
>
>
>
> Say that you forgot to take into account in your software the fact that
> every 100 years bissextile years are not as every 4 years, you will never
> find it whatever the number of kilometres, cars and hours you use the
> system between 2001 and 2099…
>
>
>
> And whatever the good performance of your system during 99 years, there
> will be absolutely zero excuse for the consequent accidents …
>
>
>
> A good way to challenge the designers of such systems would be to make
> their children responsible for the damages …
>
>
>
> Bertrand Ricque
>
> Program Manager
>
> Optronics and Defence Division
>
> Sights Program
>
> Mob : +33 6 87 47 84 64
>
> Tel : +33 1 58 11 96 82
>
> Bertrand.ricque at sagem.com
>
>
>
> *From:* systemsafety [
> mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de
> <systemsafety-bounces at lists.techfak.uni-bielefeld.de>] *On Behalf Of *Nick
> Tudor
> *Sent:* Thursday, April 21, 2016 2:27 PM
> *To:* Matthew Squair
> *Cc:* Bielefield Safety List
> *Subject:* Re: [SystemSafety] How Many Miles of Driving Would It Take to
> Demonstrate Autonomous Vehicle Reliability?
>
>
>
> This report has just come to my attention.  Stats based and an interesting
> read as it addresses most of the points made on this thread in one way or
> another:
>
>
>
> http://www.rand.org/pubs/research_reports/RR1478.html
>
>
> Nick Tudor
>
> Tudor Associates Ltd
>
> Mobile: +44(0)7412 074654
>
> www.tudorassoc.com
>
> <image001.jpg>
>
>
>
> *77 Barnards Green Road*
>
> *Malvern*
>
> *Worcestershire*
>
>
> *WR14 3LRCompany No. 07642673*
>
> *VAT No:116495996*
>
>
>
> *www.aeronautique-associates.com <http://www.aeronautique-associates.com>*
>
>
>
> On 18 April 2016 at 22:01, Matthew Squair <mattsquair at gmail.com> wrote:
>
> More that I don't see the value of multi million trip test programs that
> others might. ;)
>
> Matthew Squair
>
>
>
> MIEAust, CPEng
>
> Mob: +61 488770655
>
> Email; Mattsquair at gmail.com
>
> Web: http://criticaluncertainties.com
>
>
> On 18 Apr 2016, at 10:13 PM, Peter Bernard Ladkin <
> ladkin at rvs.uni-bielefeld.de> wrote:
>
>
>
> On 2016-04-18 14:03 , Matthew Squair wrote:
>
> But I'd personally be comfortable after a couple of months of realistic
> road trials.
>
>
> Hey, folks, we gotta volunteer!......... How you gonna line all those
> companies up, Matthew? :-)
>
> PBL
>
> Prof. Peter Bernard Ladkin, Faculty of Technology, University of
> Bielefeld, 33594 Bielefeld, Germany
> Je suis Charlie
> Tel+msg +49 (0)521 880 7319  www.rvs.uni-bielefeld.de
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>
>
>
> <image002.jpg>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> Virus-free. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> Virus-free. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <systemsafety at techfak.uni-bielefeld.de>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> <systemsafety at techfak.uni-bielefeld.de>
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> Virus-free. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
>
>
> --
>
> *Matthew Squair*
>
> BEng (Mech) MSysEng
>
> MIEAust CPEng
>
>
>
> Mob: +61 488770655
>
> Email: MattSquair at gmail.com
>
> Website: www.criticaluncertainties.com <http://criticaluncertainties.com/>
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>

-- 
*Matthew Squair*
BEng (Mech) MSysEng
MIEAust CPEng

Mob: +61 488770655
Email: MattSquair at gmail.com
Website: www.criticaluncertainties.com <http://criticaluncertainties.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20160423/26fc9337/attachment-0001.html>