[SystemSafety] Fwd: Re: CbyC and unit testing

[Olwen Morgan]

On 04/07/2020 22:10, Martyn Thomas wrote:
>
> "What do you do when your system *fails* a UT?".
>
I check for errors in the code *then in the tooling system, then 
backwards through the process following the traceability trail (assuming 
it hasn't been eaten by the creatures in the forest)*.

My reason for this is that part of my professional background is in 
compiler testing for which I have held registration as a UKAS (successor 
to NAMAS) assessor. Under the old NAMAS standard an assessor is required 
to check test systems for influence factors (a term used in metrology) 
that could affect the results of testing or measurement. Among these 
factors are defects in the testing and measurement systems used.

Owing to this concern, I have produced designs for compiler test control 
systems that run self-check routines after each run of a program in a 
compiler validation suite. Among other things, such self-tests serve the 
purpose of guarding against the test control system, among other things:

(1) failing to satisfy assumptions upon which the integrity of testing 
relies,

(2) falling into a state such that the integrity of testing is compromised,

(3) being unable to re-establish for rerun, the test conditions under 
which a test may have failed,

(4) failing adequately to control other influence factors within its 
control that might, if uncontrolled, render the results of testing 
untrustworthy.

I am not aware of any current compiler validation suite that comes with 
a test control system that seeks to achieve this degree of control. Few 
compiler test suite vendors seem to take much notice of the relevant 
international standard, ISO/IEC 17025:2017 General requirements for the 
competence of testing and calibration laboratories (which UKAS now uses 
instead of the old NAMAS standard of similar title). Indeed, I have 
actually wanted to abandon some attempts at compiler validation because 
it proved impossible to set up an adequately self-checking test control 
system.

What price CbyC reliance on compilers that may not have been constructed 
using CbyC and may not have been tested under adequately controlled 
conditions?

(Takes deep breath and waits for Derek Jones, who AFAI am aware has 
never been a UKAS assessor, to hint, not for the first time, that I'm 
anally obsessive about compiler validation.)


Counter question:

If you are using CbyC and a system unit fails under test at a stage of 
testing later than UT, what do you do?


Olwen





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20200705/fda1e353/attachment.html>