[SystemSafety] MC/DC coverage assumptions

[Tom Ferrell]

Assuming the tool is reporting coverage metrics correctly (something that should always be confirmed through tool qualification), then anything less than 100% means the following:

1.	Missing test case - determine if one can be added within the current test structure and is appropriate given the stated requirements (at least with DO-178x, all tests are supposed to be requirements driven.
2.	If missed coverage relates to functionality for which no requirement exists, then the answer may be, add the requirement and then the subsequent test OR
3.	Missed coverage is perhaps for defensive code (e.g., when others in a switch statement) for which requirements might not exist but a coding standard required.  In some cases this can be analyzed and a rationale for the coverage gap documented with the verification results (e.g., the above mentioned when others when all cases are accounted for by a test covering all possible input values to the switch).  Other missed cases may relate to deactivated code for which additional rules apply (e.g., showing the deactivation mechanism works correctly).
4.	Finally, there may be dead code that needs to be removed.  This has become less of a problem over time with the use of smart compilers, but it is a possible cause of gaps.  In general, dead code should be removed.

As Michael noted, this is all pretty well explained in both section 6 of DO-178C and DO-248C (various FAQs and discussion papers). 

-----Original Message-----
From: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of Derek M Jones
Sent: Wednesday, February 28, 2018 11:10 AM
To: systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] MC/DC coverage assumptions

Steve,

> ³The MC/DC coverage numbers were a lot better than the statement and 
> branch coverage.  This is obviously a mistake, at best they can be as 
> good as.²
> 
> What does ³coverage number² mean here? More defects revealed? More 
> test cases required?

The number of functions having a given percentage coverage.
The paper had almost all functions with MC/DC at 100% (which implies statement coverage for those functions is 100%, which it was not).


> -----Original Message-----
> From: systemsafety 
> <systemsafety-bounces at lists.techfak.uni-bielefeld.de>
> on behalf of Derek M Jones <derek at knosof.co.uk>
> Organization: Knowledge Software, Ltd
> Date: Wednesday, February 28, 2018 at 6:35 AM
> To: "systemsafety at lists.techfak.uni-bielefeld.de"
> <systemsafety at lists.techfak.uni-bielefeld.de>
> Subject: [SystemSafety] MC/DC coverage assumptions
> 
> All,
> 
> I was recently reading a paper that compared unit testing of 
> industrial embedded software with some open source programs.
> The comparison included a table of statement, branch and MC/DC 
> coverage, items in the table included: aerospace software, automotive 
> software and subway signal software
> 
> The MC/DC coverage numbers were a lot better than the statement and 
> branch coverage.  This is obviously a mistake, at best they can be as 
> good as.
> 
> I emailed the authors, who have been very prompt replying.
> The latest reply was a bit surprising.
> 
> The algorithm they used for MC/DC assumes that a function containing a 
> single branch (e.g., an if-statement with no else part) and the test 
> involves a single condition (i.e., no AND or OR conditions), then 100% 
> MC/DC coverage is assumed, even if 100% branch coverage is not 
> obtained.
> 
> Sounds like a mistake in their algorithm.  However, they claim there 
> is some amount of existing practice and even call out Testbed as 
> behaving like this (I don't have a copy to check this out).
> 
> Somebody please tell me that this is not an assumption made by 
> commercial packages when calculating MC/DC coverage.
> 
> The authors admit that MC/DC coverage cannot be better than statement 
> and branch coverage, and admit the current presentation of MC/DC 
> coverage in the table could be misleading.  They are going to release 
> a version with corrected data.
> 

-- 
Derek M. Jones           Software analysis
tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com
_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE