[SystemSafety] OpenSSL Bug

[Steve Tockey]

David,

"Probably because the whole discussion started from a bug (in OpenSSL)
that precisely lies in those 7%. :-)"

Actually I'm going to respectfully disagree with that. I'm going to claim
I can trace it back to incomplete requirements & resulting inadequate
design. A parameter on a function call was missing a range check. Why?
Because there wasn't an explicit requirement that the value represented in
that parameter be within a specific range. Had there been an explicit
requirement, and had the designer/programmer paid attention to that
requirement then they would have needed a range checking 'if' statement to
satisfy that requirement.

And even if there were no explicitly stated requirement, had the developer
had any brains and used Design-by-Contract then they would surely have
noticed that the value range of that parameter's data type was much bigger
than what the function was supposed to be able to handle. The need for a
range check would have been bloody obvious.

It wasn't at all a 7% defect, it was an 83% defect.


"More seriously, I would make the distinction between two kinds of
approaches: ..."

Again, slight disagreement. I don't know the exact source of the defect
injection data (I.e., what specific kinds of projects the data came from).
However, I do know (of) the person who published the data: James Martin.
Knowing (of) Mr. Martin, I'm hard pressed to say that the data came from
any safety critical projects.

Besides that, my own experience with both safety-critical and non-
development projects doesn't show any appreciable difference in the defect
*injection* statistics. The big difference, IMHO, is that the
safety-critical projects make much better use of reviews and the defect
*detection* statistics are heavily skewed towards being found and fixed
earlier.


"Thus I have a practical question: suppose I have a company that lacks
such good practices (at least for the review or safe coding rules one).
I'm not a project manager, just a low level software engineer working on
a given project. How can I help improve the development process to reach
such level of good practices?"

Contact me directly, I'll send you a paper called "How Healthy is your
Software Process?". By instrumenting the project's defect tracking system
with very simple data we can easily show that the majority of defects are
requirements & design injected and that more than half of the
organization's capacity to do development work is being wasted in finding
and fixing mistakes that were made much earlier. Managers have a tendency
to discount what the techies are telling them, but it's hard to ignore the
data when it's staring them in the face.

If anyone wants the paper, I'm happy to share it.


-- steve




-----Original Message-----
From: David MENTRE <dmentre at linux-france.org>
Date: Wednesday, April 16, 2014 6:32 AM
To: "systemsafety at lists.techfak.uni-bielefeld.de"
<systemsafety at lists.techfak.uni-bielefeld.de>
Subject: Re: [SystemSafety] OpenSSL Bug

Hello,

Le 16/04/2014 13:23, Steve Tockey a écrit :
> No offense inferred, but thanks for pointing it out.

The same for me. I should have kept the classical "if (a == SUCCESS)"
example. Another example: C language does not forces you to check the
return value of a function (that might flag an error case). Most of
other languages I know would force you to check it (or at least issue a
warning).

> Besides, echoing this and Derek M. Jones' last post, I think this whole
> discussion is focusing on relatively minor issues and completely missing
> the big picture.
[...]
> the simple fact is that 83% of
> all defects exist before a single line of code was ever written. Why
> aren't we attacking the 83%, not the 7%???

Probably because the whole discussion started from a bug (in OpenSSL)
that precisely lies in those 7%. :-)

More seriously, I would make the distinction between two kinds of
approaches:

  1. People that produce software following the good practices you
describe below, probably to satisfy DO-178C or EN 50128 standards. In
such case, their error figures are probably in the above range (83%
requirement + design, 7% code).

  2. People that produce software *without* following proper practices,
probably like OpenSSL developers and most companies developing software
in non-safety critical domain. In that case the coding errors are much
more prevalent. And we are several people on this list to think that
using better tool could help catch more of these errors. I do agree that
following good practices could have similar effect, but the simple fact
is that they are not doing it, probably due to lack of knowledge, time,
money or because this is not a paid-for job. You might counter-argue
that if they are not using good practices, they won't use static
analysis tool neither. You might be right. ;-)

> A
> team with good requirements, design, and review practices can write good,
> safe code in any language, including C.

I agree.

Thus I have a practical question: suppose I have a company that lacks
such good practices (at least for the review or safe coding rules one).
I'm not a project manager, just a low level software engineer working on
a given project. How can I help improve the development process to reach
such level of good practices?

Sincerely yours,
david


_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE