[SystemSafety] AI and the virtuous test Oracle - action now!

[Prof. Dr. Peter Bernard Ladkin]

On 2023-07-26 08:52 , Dr. Brendan Patrick Mahony wrote:
> 
>> [PBL] Am I right that this is a regime that concerns cybersecurity? And that it is a US
>> Government process? And that it concerns largely IT systems?
> 
> All good questions I was hoping someone wiser might have answers to.

OK; I'll turn my first two into assertions. It seems to me that ATO/cATO concerns just 
cybersecurity. It also seems to me that ATO/cATO appears to be a requirement for USG IT operations.

Many governments now have cybersecurity requirements for their operations. In the UK there is the 
Cyberessentials scheme, which has been running for well over a decade. I can't talk about ATO but I 
can talk a bit about Cyberessentials. NSCS notes that Cyberessentials certification is required for 
*some* government contracts. I recall there was a lot of discussion about the scheme at the 2016 IET 
System Safety and Cyber Security conference in London.

I have some passing acquaintance with Cyberessentials, but I have personally neither self-certified 
nor been through certification. It started out as thought to be a "basic" scheme, but I had a 
knowledgeable colleague who had spent a couple of years trying to self-certify his consultancy, and 
there was also a lot of very odd talk at the 2016 conference from certification organisations who 
were disparaging their customers (that is not a business practice I would recommend for anyone, let 
alone at a technical conference....). The NCSC recognised at the time that Cyberessentials was not 
as easy as HMG initially intended (the programme is in fact older than NCSC). The cybersecurity 
officer for a major multinational told me that Cyberessentials certification was one of the most 
expensive internal exercises the company had ever undertaken.

There is nothing to criticise here. It is that cybersecurity "basics" turn out to be not as easy as 
people thought, and don't completely work. If people ask for advice to inhibit burglary, they will 
be told "have good locks on all your doors and locking devices on your accessible windows and use 
them." Simple and good advice. But it has been around for years and burglary has not ceased.

The major issue with IT cybersecurity (not just with IT) is the supply chain. Bruce Schneier opines 
that it is an insoluble problem. Supply chain issues are well illustrated in USG IT circles by the 
Solarwinds incident.

I think ATO/cATO concerns IT, but I ask for correction if this is mistaken.

IT means systems whose useful outputs are information, or information structures, displayed largely 
on screens when they are displayed. None of these are safety-related under the IEC definition of 
functional safety. Functional safety largely concerns operational technology (OT) - control systems.

I would say that the difference between IT cybersecurity and OT cybersecurity is quite significant, 
because the general difference in architectures is quite significant. Detection is generally easier 
in OT: it is generally easier to spot that a control system isn't doing what the operator thinks it 
should be doing, and to invoke shutdown or other prophylaxis, as (in IT) to detect somebody reading 
all your confidential records, or diverting a few extra pence of a transaction somewhere where it 
shouldn't be going. (Financial transaction systems have traditionally been considered IT. Since they 
perform transactions it is possible to consider them in principle as OT, but here "harm" would be 
interpreted as financial error, which is not in the IEC definition.) OT of necessity has system 
architectures which are more clear than those of large complex IT systems.

The difference between inhibiting/mitigating something (cybersecurity incidents) and preventing 
something (accidents) is also much more than a matter of degree.  A HazAn performed on a large IT 
system for cybersecurity hazards will turn up masses of potential vulnerabilities, most of which the 
operator just has to swallow/"learn to live with" (or cease operations); whereas for safety hazards 
each and every one of the hazards with severity above the tolerable risk level *must* be avoided or 
mitigated as a matter of law (this is how ALARP, a legal requirement, plays out in common-law 
countries).

> My concerns are that it seems to be being promoted as an approach to in-servicing “intelligent” 
> systems, meaning they intend to use cATO to authorise the fielding of whole swathes of “operator” 
> replacement software.
Certainly something worthy of discussion. If no one here answers, I'll see if I can find pointers 
elsewhere.

PBL

Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de