[SystemSafety] Recent over-the-air update improved the car’s braking

[Martyn Thomas]

Bev

I'm an enthusiast for testing.

I'm an even greater enthusiast for drawing valid conclusions from
passing the tests and for taking the appropriate action if tests fail.

I believe that it is easier to draw valid conclusions if you have
decided and documented what you are trying to prove or disprove by your
testing (aka the scientific method) and if you can show that the design
of your tests is able to deliver that.

Without this, passing a set of tests tells you very little and may
engender ill-founded confidence, whereas failing some tests tells you
that you screwed up somewhere earlier in your development and that you
had better figure out where and why so you can reduce the risk of it
happening again.

Martyn


On 31/05/2018 16:24, Littlewood, Bev wrote:
> Hi Martyn
>
> I have some sympathy with Derek’s view, but I’d put it a bit more
> strongly. Rather than his “warm feeling” from /some/ testing, I'd
> prefer there to be “sufficient” testing - i.e. to be able to make a
> claim about its reliability/safety with a high enough confidence.
> Which I think is your implicit point: how much is “sufficient”?
>
> Without going into the numbers (which always seems to be like poking a
> hornets’ nest on this forum) I think we could all agree that in the
> time that elapsed only very weak claims from testing could be made (if
> indeed they did any testing at all).
>
> I suppose they might say that they “just” tweaked some parameters,
> with the implication that there were not substantive changes to the
> software. So that the results from any extensive software testing they
> had done in the past (if indeed they did this) could still be relied
> upon. But this won’t wash, will it? We would need to know how
> extensive the parameterisation is here, and whether all possible
> parameterisations had been tested to support the required
> reliability/safety claims. 
>
> Many years ago I was involved in the controversy surrounding the
> safety claims for the software-based Primary Protection System of the
> Sizewell B nuclear reactor here in the UK. The “software” was a couple
> of hundred thousand lines of code. But the “configuration data” was of
> comparable size. Testing the system in just one configuration was
> problematic. Gaining confidence in its safety in /all/ configurations
> was very hard.
>
> I wonder whether this is an issue in the Tesla case? Seems to me it
> might be.
>
> Of course, an important difference in these situations is that in the
> Sizewell case a /single/ nasty event needs to be highly improbable. In
> the Tesla case, maybe they (we?) can live with a couple of nasty
> events. After all, if this happens, they’ll fix the problem, won’t
> they? I.e. they’ll tweak it after a few incidents (maybe deaths). But
> we would have the same sort of doubts about the efficacy of this tweak
> as we have had in the current one. And so on... 
>
> Cheers
>
> Bev
>
>> On 31 May 2018, at 09:07, Martyn Thomas
>> <martyn at thomas-associates.co.uk
>> <mailto:martyn at thomas-associates.co.uk>> wrote:
>>
>> What specific objectives should such testing have? How much and what
>> sort of testing would deliver the required evidence that the objectives
>> had been achieved?
>>
>> Martyn
>>
>>
>> On 31/05/2018 01:35, Derek M Jones wrote:
>>> All,
>>>
>>> No mention of testing:
>>> https://www.consumerreports.org/cars-tesla-model-3-gets-cr-recommendation-after-braking-update/
>>>
>>>
>>
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>> <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>
> _______________________________________________
>
> Bev Littlewood
> Professor of Software Engineering
> Centre for Software Reliability
> City, University of London 
> EC1V 0HB
>
> Phone: +44 (0)20 7040 8420  Fax: +44 (0)20 7040 8585
>
> Email: b.littlewood at csr.city.ac.uk <mailto:b.littlewood at csr.city.ac.uk>
>
> http://www.csr.city.ac.uk/
> _______________________________________________
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20180531/1a0e6f16/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20180531/1a0e6f16/attachment.sig>