[SystemSafety] Thorough Test Suite?

Martyn Thomas martyn at thomas-associates.co.uk
Thu Nov 14 18:29:01 CET 2019


Dewi

Yes, that's the paper. It's unfortunate that Andy German drew the
conclusions he did, if the data was as poor as you say. Why on earth did
he say that? Shouldn't the paper (which I think is still available on
the www.stsc.hill.af.mil/Crosstalk website) be withdrawn and a
correction published?  It would seem to be grossly misleading - it
certainly misled me.

Martyn

On 14/11/2019 17:11, Dewi Daniels wrote:
> Martyn,
>
> I assume the Boscombe Down analysis you're referring to is Andy
> German's CrossTalk article on Lockeed C-130J?
>
> I helped conduct the static analysis on C-130J when I was at Lloyd's
> Register. QinetiQ's analysis is flawed. Due to timescale pressures, we
> were asked to conduct the static analysis before the code had been
> tested, so I don't see that you can draw any conclusions about the
> efficacy (or otherwise) of the DO-178B verification process. Also, we
> reported every defect we found, from defects that contributed to a
> hazardous failure condition to spelling mistakes in documentation.
>
> I personally conducted the static analysis of the worst program cited
> in Andy German's paper (two software lines of code per anomaly). It
> never stood any chance of getting through a DO-178B verification
> process. The design didn't match the requirements and the code didn't
> match the design. The software was rejected by the Designated
> Engineering Representative (DER) and Lockheed eventually replaced it
> with software from a different supplier.
>
> The best program (250 lines of code per anomaly) was specified
> formally using Parnas Tables, was written in SPARK and verified
> formally using the SPADE Automatic Simplifer and the SPADE Proof Checker.
>
> Yours,
>
> Dewi Daniels | Director | Software Safety Limited
>
> Telephone +44 7968 837742 | Email d
> <mailto:ddaniels at verocel.com>ewi.daniels at software-safety.com
> <mailto:ewi.daniels at software-safety.com>
>
> Software Safety Limited is a company registered in England and Wales.
> Company number: 9390590. Registered office: Fairfield, 30F Bratton
> Road, West Ashton, Trowbridge, United Kingdom BA14 6AZ
>
>
>
> On Thu, 14 Nov 2019 at 16:31, Martyn Thomas
> <martyn at thomas-associates.co.uk
> <mailto:martyn at thomas-associates.co.uk>> wrote:
>
>     Derek
>
>     What's a "thorough test suite" and how would you recognise one? We
>     know
>     from theory and from a Boscome Down analysis that even MCDC
>     testing can
>     leave very high defect densities.
>
>     Martyn:
>
>     > A common research use (not much used in industry) is using mutated
>     > code to check the quality of a test suite, i.e., a thorough test
>     > suite will detect the added mistake.
>
>     _______________________________________________
>     The System Safety Mailing List
>     systemsafety at TechFak.Uni-Bielefeld.DE
>     <mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
>     Manage your subscription:
>     https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20191114/19ec806a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20191114/19ec806a/attachment.sig>


More information about the systemsafety mailing list