[SystemSafety] Thorough Test Suite?

[Martyn Thomas]

Derek

When you say "a "thorough test suite" is one that kills all mutants (one
does not simply detect mutants).", would that be "all the mutants we
tried" or "all possible mutants"?


For the MCDC data, I was thinking of Andy German's paper "Software
Static Code Analysis Lessons Learned" in Crosstalk, November 2003. One
of his observations is:

 0.

    It is worth reiterating that when comparing RTCA DO-178B [8] Levels
    A and B code, no discernible difference was found by static code
    analysis demonstrating that static code analysis is something you
    carry out in addition to testing. Even the most extensive testing
    does not remove the anomalies found by static code analysis.
    Surprising amounts of dead code have been found in code developed to
    RTCA DO-178B Levels A and B.

 Densities found and reported ranged from 2 LoC per anomaly to 250 LoC
per anomaly.

Martyn

On 14/11/2019 16:59, Derek M Jones wrote:
> Martyn,
>
>> What's a "thorough test suite" and how would you recognise one? We know
>> from theory and from a Boscome Down analysis that even MCDC testing can
>> leave very high defect densities.
>
> The mutation analysis people will say that a "thorough test suite"
> is one that kills all mutants (one does not simply detect mutants).



>
> MCDC testing appears to be the most thorough (and most expensive)
> approach in regular use.
>
> High defect densities?  Is this where two defects is to high?
>
> Data on non-trivial uses is remarkable hard to come by.
>
> I'm currently reworking the reliability chapter of my
> evidence-based software engineering book (draft pdf and
> data here: http://www.knosof.co.uk/ESEUR/).
>
> If anybody knows of non-trivial MCDS testing data, that is
> publicly available, lease let me know.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20191114/74b5bca7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20191114/74b5bca7/attachment.sig>