[SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

Wed Jul 3 09:16:31 CEST 2013

Steve

I'm in furious agreement with all your points. I've spent a 38 year career
making these arguments to many companies. But I also agree with Tom DeMarco
and his famous quote: "Quality is free but only if you are willing to pay
dearly for it." To be explicit about my statement: quality software is
expensive relative to the expectations of the people who allocate the
capital. Unfortunately most of these people have never been up to their
elbows in crap software and seen the disgusting waste that it triggers. They
therefore seem quite happy to allocate capital to fixing a problem but are
reluctant to spend money on preventing one. The pitch: "... give me money so
I can have requirements reviews, design reviews, code reviews, an
independent V&V group, an adequately sized test team, time to test properly
including retesting after bug fixes, an automated test environment, a
realistic schedule that accounts for all this ..."  As you probably well
know, usually attracts the response: "... so what's my return on
investment?". And the classic answer to that one is:" Well sir, nothing bad
will happen until you."  I've lived through this scenario more times than I
care to remember. It makes you feel like the little guy in this picture:
http://www.chambers.com.au/services/mentoring/mentoring.php .

Unfortunately for someone who's never seen as a disaster or thinks rolling
disaster is normal, this is not a very convincing argument. 

To compound the problem there are situations in which a tiger team of very
good people can produce a high-quality software product with none of the
above. Management loves to pounce on these anecdotes, ignoring the reality
that they are personality and situationally dependent and do not scale. I
have lived through this scenario. I was once on the management team on a
small company that built weigh bridge automation systems. Our average
project was $50,000. As a result of some bad experiences we decided to clean
up our act. Documentation was created where none existed, testing was done
where it was previously ad hoc. Then we found that we had a problem getting
out of bed for less than $100,000. Our traditional marketplace ditched us in
favour of backyard operations staffed by tiger teams that could do a similar
job for half the price. This is a good news story though. We found we could
take on larger projects and became more attractive to larger companies with
bigger budgets, so our profits increased (standby for an article on this
subject).

In general I live in hope that this situation will improve as management
matures and the people who are writing code today move up to senior
management positions. I actually saw this happen in one very large American
corporation. In the meantime I think the only solution is to do what you're
doing: get the numbers, make a business case and make it defensible so the
powers that be have no choice. Cross-cultural education is also a great
idea. I recall Nancy Leveson tried this once. She went down to the Harvard
Business School to lecture on safe software (was it to some merchant
bankers?). They were very rude to her. The experience may have scarred her
for life. (Could you tell that story again Nancy?).

Ah well - all we can do is collect the numbers, keep the faith and maintain
the rage.

Cheers

Les

From: Steve Tockey [mailto:Steve.Tockey at construx.com] 
Sent: Wednesday, July 3, 2013 2:43 PM
To: Les Chambers
Cc: martyn at thomas-associates.co.uk; systemsafety at techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring
Software]

Les,

Google is an interesting "special case", IMHO. Specifically, how would you
even know if Google gave you the right answer or not? You enter search
terms, back comes some set of pages. But are those the RIGHT pages? How
would one even know?

I doubt anyone's life is on the line over Google, Facebook, Twitter, etc and
that's definitely a good thing. The number of times that Facebook completely
crashes on my iPhone and iPad is astounding. The FB Mobile team are rapidly
approaching the top of my "World's Worst Programmers" list.

"You just can't get away from it: high-quality software is expensive."

This is where we probably disagree. If you read that "How Healthy is your
Software Process" paper, you will see that the #1 cost and schedule driver
on a typical project is rework (waste), and that it is normally bigger than
all other project cost and schedule drivers ***combined***. We actually
measured the R% in one development organization that had about 350
programmers. The as-measured R% was 56.9%. Bluntly, for every 1000 hours
those 350 developers worked 569 hours were fixing defects that had been made
earlier. Only 431 hours out of every 1000 were actual progress toward
finishing the project.

I don't have the absolute data to prove it (because the projects were done
before the idea of R% was brought out just a few years ago), but I'm
convinced that if we could have measured R% on several of my past projects
that it would be no greater than 10%. So if my project and some other
project have the same amount of  "work" to do, but my R% is only 10% and the
other projects R% is closer to 60%, then my project will deliver the same
amount of functionality in about half the time and for about half the cost.

Seriously, as Phillip Crosby said, "Quality is Free". While Crosby was
referring specifically to manufacturing organizations, his basic premise
that "there is so much waste in typical organizations that improvements in
up-stream quality not only make for better products, but cost and schedule
go down at the same time". I've seen it, it's real.

I have no disagreement with the statement that "Software is expensive". But
in my experience "high quality software is actually cheaper than typical
software".

-- steve

From: Les Chambers <les at chambers.com.au>
Date: Tuesday, July 2, 2013 5:27 PM
To: Steve Tockey <Steve.Tockey at construx.com>
Cc: "martyn at thomas-associates.co.uk" <martyn at thomas-associates.co.uk>,
"systemsafety at techfak.uni-bielefeld.de" <systemsafety at techfak.uni-bielefeld.
de>
Subject: RE: [SystemSafety] Qualifying SW as "proven in use" [Measuring
Software]

Steve

Code evaluation can sometimes be possible in unit level testing when you are
dealing with smaller code bodies, especially if you have automated tools. Of
course it should be done in the development environment. I agree with you
that it's impossible in integration and system level testing. I heard an
extreme case the other day: a Google tester related that, in his development
career, he was making a small change to some server software when he noticed
there were more than 100,000 artefacts in the build. Companies like Google
face extreme real-world conditions. The same man went to the managers of
Google server farms and laid out his machine requirements into the future
for testing. He said, "These guys were pretty blunt. They said fine, you can
have all that machine time but we'll have to shut Google down to give it to
you." So the test team came up with an innovative solution: they tested
every second mod. I hope nobody's life is on the line over a Google query.

You just can't get away from it: high-quality software is expensive.

Les

From: Steve Tockey [mailto:Steve.Tockey at construx.com] 
Sent: Tuesday, July 2, 2013 8:47 AM
To: Les Chambers
Cc: martyn at thomas-associates.co.uk; systemsafety at techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring
Software]

Les,

I think that sounds good in theory but it may not work effectively in
practice. The issue is that almost all of the test teams I know don't have
inside (aka "white box") knowledge of the software they are testing. They
are approaching it purely from an external ("black box") perspective. They
can't tell if the code has high cyclomatic complexity or not.

In principle, testers should be given the authority to reject a piece of
crap product. That's their job in all other industries. But in software
(non-critical, mind you), testing is usually a window dressing that's mostly
overridden if it meant threatening promised ship dates.

-- steve

发自我的 iPad

On Jul 1, 2013, at 2:36 PM, "Les Chambers" <les at chambers.com.au> wrote:

Steve

One way to achieve this is to empower test teams. Management issues an
encyclical that: SOFTWARE IS NOT A GIVEN. That is, "If it's too complex to
test effectively - reject it. Don't waste your time composing feckless tests
for crap software. Send it back to it heathen authors.

Kill it before it gets into production.

Les

On 02/07/2013, at 3:16 AM, Steve Tockey <Steve.Tockey at construx.com> wrote:

Martyn,

My preference would be that things like low cyclomatic complexity be
considered basic standards of professional practice, well before one even
started talking about a safety case. Software with ridiculous complexities
shouldn't even be allowed to start making a safety case in the first place.

-- steve

From: Martyn Thomas <martyn at thomas-associates.co.uk>
Reply-To: "martyn at thomas-associates.co.uk" <martyn at thomas-associates.co.uk>
Date: Monday, July 1, 2013 10:04 AM
Cc: "systemsafety at techfak.uni-bielefeld.de"
<systemsafety at techfak.uni-bielefeld.de>
Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring
Software]

Steve

It would indeed be hard to make a strong safety  case for a system whose
software was "full of defects". 

High cyclomatic complexity may make this more likely and if a regulator
wanted to insist on low complexity as a certification criterion I doubt that
few would complain. Simple is good - it reduces costs, in my experience.

But if a regulator allowed low complexity as a evidence for an acceptably
low defect density, as part of a safety case, then I'd have strong
reservations.  Let me put it this way: if there's serious money to be made
by developing a tool that inputs arbitrary software and outputs software
with low cyclomatic complexity, there won't be a shortage of candidate tools
- but safety won't improve. And if you have a way to prove, reliably, that
the output from such a tool is functionally equivalent to the input, then
that's a major breakthrough and I'd like to discuss it further.

Martyn

On 01/07/2013 17:18, Steve Tockey wrote:

Martyn,

"The safety goal is to have sufficient evidence to justify high
confidence that the software has specific properties that have been
determined to be critical for the safety of a particular system in a
particular operating environment."

Agreed, but my fundamental issue is (ignoring the obviously contrived
cases where the defects are in non-safety related functionality) how could
software--or the larger system it's embedded in--be considered "safe" if
the software is full of defects? Surely there are many elements that go
into making safe software. But just as surely, IMHO, the quality of that
software is one of those elements. And if we can't get the software
quality right, then the others might be somewhat moot?

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20130703/0399a804/attachment-0001.html>