[SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

Thu Jun 20 11:31:54 CEST 2013

Dear all,

I would like to have your opinion on the following question.

Why do all these nice and fancy tools, such as SQALE and many others, are only available for ADA and C++ and not for the IEC61131-3 languages used in process and manufacturing industries ?

Is it because:
* These languages are not considered sufficiently "seriously safe" by "serious persons" to deserve academic works and tools ?
* It is not possible to access these languages because they are proprietary artefacts not open to academic works ?
* The designers and suppliers of these languages don't allow access to the academic and software community ?
* The academic and software community don't identify the need ?
* The final intended users are not sensitive to theissue ? Competent enough to adress the issue ?
* Any work on the languages is useless because the OS below is even more unknown and not accessible ?
* Anything else ?

Bertrand RICQUE
Program Manager, Optronics and Defense Division

T +33 (0)1 58 11 96 82
M +33 (0)6 87 47 84 64
23 avenue Carnot 
91300 MASSY - FRANCE 
http://www.sagem-ds.com

-----Original Message-----
From: systemsafety-bounces at techfak.uni-bielefeld.de [mailto:systemsafety-bounces at techfak.uni-bielefeld.de] On Behalf Of Thierry.Coq at dnv.com
Sent: Thursday, June 20, 2013 11:19 AM
To: ladkin at rvs.uni-bielefeld.de; systemsafety at techfak.uni-bielefeld.de
Cc: denis.chalon at automationsquare.com; Jean-Louis.Letouzey at inspearit.com
Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

Hi Peter,

To answer your questions:
1°) Yes, there is some objective evidence that there is a correlation between a low SQALE index and quality code. For example ITRIS has conducted a study where the "good quality" code is statistically linked to a lower SQALE index, for industrial software actually used in operations.

No, there is not enough evidence, we wish there would be more people working on getting the evidence.

But more than that, SQALE IS a quality model: it DEFINES what is "good quality" code. It defines objective, measurable REQUIREMENTS that can be achieved, and that can be objectively measured between vendors and owners. 
If a given check point is later found to be not relevant, there is no problem to discard the point.

2°) For high-critical software. Yes, SQALE has been used on high-critical software, where it basically automates the (pre-)defined quality rules. It is a useful tool to help what you call "experienced critical-system programmers". 
There is a published paper by Jean-Pierre Rosen and myself on ADA open-source code where you can find more information: we've used open source because most applications of SQALE in industry are confidential.

In addition, we're always looking for clear, unambiguous (automatable), without exceptions, checkpoints. For example, one of our current checks points is a higher limit of cyclomatic complexity V(G) of 15 and 60 for one operation (method/procedure/function/whatever you call it). The V(G) can be measured by many tools today, or approximated to an extent. The usual limit in the literature is around 10 (for example, here: http://www.sei.cmu.edu/reports/97hb001.pdf). The thresholds in SQALE are high in order to reduce discussions about real-life implementations. Our measurements show that most operations fall within this limit
But there are challenges to this check point. For example:
a) V(G) actually measures size and not complexity. This fact is used in SQALE as it uses the aggregate V(G) to compute densities (very interesting, btw). SQALE does not have any checkpoint on aggregate V(G), only on individual operations,
b) some operations are complex and breaking them down is "artificial". SQALE's approach is to require breakdown as a DESIGN decision. It's possible in some implementations of SQALE either to require a rewrite, or to have a review and flag the particular operation as having an "accepted non-compliance". The last approach is to accept high V(G) in an operation if it follows a particular pattern: the whole V(G) originates from one "CASE ... OF" or "SWITCH" construct. However a mix of "CASE OF", "IF" "WHEN" and other constructs would result in a non-compliance.

Another example is copy/paste. SQALE detects a non-compliance for identical code above 100 tokens. This is about 5-15 lines of code, depending on the language, IDENTICAL in 2 or several areas. What we have observed is that there are clear differences between projects. Some projects tend to have large copy/paste ratios, while other projects (and the two we analysed in our ADA paper) have really low copy/paste. We do know which has the better quality. Again we get challenges on the usefulness of copy/paste, although most quality plans identify this as a non-compliant practice:
a) copy/paste does not in fact increase the number of defects. SQALE's point of view is that copy/paste in fact increases the test effort and reduces testability, so it is  a non-compliance. Copy/paste may increase defects but the literature is not clear. It also reduces maintainability since a defect/change in one paste instance must be corrected in every instance, but SQALE's model puts testability before maintainability, so it counts copy/paste only once.
b) copy/paste on "generated code" has no sense. SQALE's point of view is that there is no reason people writing code generators should abstain from applying the quality rules applicable to hand-crafted code. In fact we would expect them to do a better job at writing code. 

Basically, SQALE is a framework to harmonize and hierarchize quality check points. Each of those check points (and their thresholds) may be challenged. But when performing assessments, the 10 worst pieces of code identified by their high aggregate SQALE index are much poorer quality, compared to the remainder of the software, as assessed by humans. You can try it yourself. Deliberately, SQALE has been made freely available so additional and independent results can be obtained. Several tool vendors have implemented SQALE to complement their measurement points, and the SQALE authors have no commercial interests with tool vendors.

Peter, any comments or suggestions for improvement coming from you would be really helpful. Or from this mail list ;-).

Thierry Coq
PS. All opinions are my own, not necessarily those of my employer.

-----Original Message-----
From: systemsafety-bounces at techfak.uni-bielefeld.de [mailto:systemsafety-bounces at techfak.uni-bielefeld.de] On Behalf Of Peter Bernard Ladkin
Sent: mercredi 19 juin 2013 17:07
To: systemsafety at techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

Thierry,

is there any objective evidence that using SQALE improves the quality of the code?

If yes, is there any objective evidence that using SQALE improves the quality of high-quality code written by experienced critical-system programmers?

If yes, where would that be found in the engineering literature?

PBL

Prof. Peter Bernard Ladkin, Faculty of Technology, University of Bielefeld, 33594 Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs.uni-bielefeld.de

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE

**************************************************************************************
The contents of this e-mail message and any attachments are confidential and are intended solely for the addressee. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Any unauthorized use, copying or dissemination of this transmission is prohibited. Neither the confidentiality nor the integrity of this message can be vouched for following transmission on the Internet.
**************************************************************************************

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#