[SystemSafety] State of the art for "safe Linux"

[Dewi Daniels]

On Wed, 7 Aug 2024 at 19:03, Steve Tockey <steve.tockey at construx.com> wrote:

>
> Paul,
> I should have jumped on this from the start …
>
> *“DO-178C doesn't require you to achieve 100% structural coverage*
> *through unit tests. It requires that your requirements-based tests*
> *(which can be hardware/software integration tests, software*
> *integration tests or low-level tests) cover all the software*
> *requirements and exercise all the code.”*
>
> That’s not at all how I read DO-178C. Yes, DO-178C does mandate what I
> would call “100% Requirements Coverage”. But in addition, Table A-7 on
> Structural Coverage Analysis clearly does mandate (quote):
>
> Row 5 "Test coverage of software structure (modified condition/decision)
> is achieved” for DAL A
> Row 6 “Test coverage of software structure (decision coverage) is
> achieved” for DAL A , B
> Row 7 “Test coverage of software structure (statement coverage) is
> achieved” for DAL A, B, C
>
> While DO-178C may not explicitly mandate that said Structural Coverage be
> demonstrated at the unit test level, it is often impossible in practice to
> achieve it in any other way.
>

What I wrote is correct. I agree that DO-178C requires test coverage of
software structure at Levels A, B and C. However, this can be achieved
through a combination of hardware/software integration tests, software
integration tests and low-level tests as shown in DO-178C figure 6-1. I
also agree that achieving structural coverage will invariably require at
least some unit tests. This is acknowledged in the note to DO-178C §6.4.1,
which reads "In many cases, the requirements-based coverage and structural
coverage necessary can be achieved only with more precise control and
monitoring of the test inputs and code execution than generally possible in
a fully integrated environment. Such testing may need to be performed on a
small software component that is functionally isolated from other software
components".

I was responding to Paul's response to my question

> Even Level C isn't that hard. Why wouldn't
> you want to achieve statement coverage?

in which he wrote

Actually, I wouldn't, because of my personal bias. I've successfully
> delivered small amounts of critical code (in the few thousands of LoC
> range) without unit tests, for systems which then worked for years
> without problems. Usually I've had more success with system tests than
> unit tests, on many projects.
>

Paul seemed to be assuming that statement coverage is achieved through unit
testing alone. That was my point in writing "DO-178C doesn't require you to
achieve 100% structural coverage through unit tests". I should have added
"alone".

In fact, “Statement Coverage” as a Structural Coverage criterion is the
> absolute weakest of all Control-Flow based Structural Coverage criteria and
> it has logical holes in it so big that one could drive a proverbial double
> decker London City bus through them.
>

Absolutely.  My original comment was "I don't understand why it's so hard
for Linux to just comply with standards such as RTCA DO-178C/EUROCAE
ED-12C? DO-178C Level D is just Software Engineering 101. Even Level C
isn't that hard. Why wouldn't you want to achieve statement coverage?". I
find it very hard to understand why it's so difficult to create a 'Safe
Linux' that complies with DO-178C Level C, let alone DO-178C Level D. As
Prof. Peter Ladkin wrote, "This business about Linux kernel for
safety-related systems has been going on for so long. Other companies have
written kernel-function OSs for safety-critical systems, and have
assessment certificates for them from recognised assessors, all within that
time. What's wrong with trying that route?".

And the vast majority of organizations I work with only aim for 60% to 70%
> Statement Coverage for even their most critical code. Shocking.
>

There's a very big gap between safety-critical software and other kinds of
software. There always has been. I don't entirely understand why.

Yours,
Dewi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20240808/826f9e9f/attachment.html>