[SystemSafety] Agile methods

Steve Tockey Steve.Tockey at construx.com
Mon Sep 2 19:50:53 CEST 2013


While we're on the topic...

Wallace & Kuhn wrote a paper titled, ³Converting System Failure Histories
into Future Win Conditions²
(http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28.2701&rep=rep1&
type=pdf)

One interesting tidbit out of the paper is (page 11), "It is significant
however, that of the 109 reports that are detailed, 98 % showed that the
problem could have been detected by testing the device with all pairs of
parameter settings." So, while it appears that all-pairs testing with
respect to device parameters is pretty effective in revealing problems, it
doesn't appear that the FDA has made any move on requiring that kind of
testing before the candidate device is approved...


-- steve



-----Original Message-----
From: Derek M Jones <derek at knosof.co.uk>
Organization: Knowledge Software, Ltd
Date: Monday, September 2, 2013 10:36 AM
To: "systemsafety at lists.techfak.uni-bielefeld.de"
<systemsafety at lists.techfak.uni-bielefeld.de>
Subject: Re: [SystemSafety] Agile methods


> Given the sad state of medical device software, it does not seem that
> simply using Agile methods proves anything. Homa Alenzadeh, a grad
>student
> at UIUC, has looked at medical device recalls. The following is from an
> abstract she wrote for a talk she gave: "Medical device incidents are one

Paper + spreadsheet listing 1,211 recall events:
http://users.crhc.illinois.edu/alemzad1/pub.html

> of the leading causes of serious injury and death in the United States.
> Between years 2006 and 2011, 5,294 recalls and approximately 1.2 million
> adverse events of medical devices were reported to the US Food and Drug
> Administration (FDA). Almost 23 percent of these recalls were due to
> computer-related failures, of which approximately 94 percent presented
> medium to high risk of severe health consequences (such as serious injury
> or death) to patients.   '
>
> If my arithmetic is correct (1,200,000*.23 and then divide by 5 then
> multiply by .94), it appears from Homa's data that there are 51,888
> reported potentially serious incidents to the FDA regarding software each
> year. That is 144 every day. And many incidents are not reported (the
> reporting rules are pretty sketchy) and that is only in the U.S.
>
> The simple fact that mdedical device manufacturers have used Agile
>methods
> does not prove anything about their efficacy for safety-critical systems
> given the sad state of safety in that industry.
>
> Nncy
>
>
> On Mon, Sep 2, 2013 at 12:40 PM, Steve Tockey
><Steve.Tockey at construx.com>wrote:
>
>>   All,
>> I have personal experience using Agile, and personal experience doing
>> safety-critical development (DO-178B), but not personal experience doing
>> agile development for a safety critical system. Instead of being
>>polarizing
>> and claiming "it can't be done", I'll simply say that I believe that it
>>is
>> possible to do it‹it could just end up being very costly to do so. The
>> worst case would be developing the code for the system in an truly agile
>> way, and then looping back and re-developing it with the necessary
>>safety
>> analysis, etc. in place using more traditional processes.
>>
>>   All that said, I have a report titled, "AAMI
>> TIR45: 2012 Technical Information Report      Guidance on the use of
>>AGILE
>> practices in the development of medical device software". AAMI is the
>> "Association for the Advancment of Medical Instrumentation" and they're
>> based at
>> 4301 N. Fairfax Drive, Suite 301
>> Arlington, VA   USA    22203-1633
>> www.aami.org
>>
>>   The report abstract states, "Over the past several years, AGILE
>>software
>> development has become an accepted method for developing software
>>products.
>> There have been questions from both manufacturers and regulators as to
>> whether (or which) AGILE practices are appropriate for developing
>>medical
>> device software. Enough medical device manufacturers have implemented
>>AGILE
>> practices in their software development so that answers to these
>>questions
>> can be documented. Having clear guidance of which practices have been
>>found
>> to be appropriate will be very useful for all developers of medical
>>device
>> software. This TIR will provide recommendations for complying with
>> international standards and U.S. Food and Drug Administration (FDA)
>> guidance documents when using AGILE practices to develop medical device
>> software."
>>
>>   A preview copy is at:
>>
>> 
>>http://marketplace.aami.org/eseries/scriptcontent/docs/Preview%20Files/TI
>>R45_1208_preview.pdf
>>
>>   I have a licensed copy, but it's copyrighted material and can't be
>> shared. If you're really interested in the content, you'll have to get a
>> copy on your own (sorry).
>>
>>   Having said that, three comments about all this:
>>
>>   1) I don't know what sort of status AAMI has with respect to FDA
>>policy
>> on medical device software development. In other words, I don't know if
>>the
>> FDA has actually agreed to this, or it's simply AAMI's suggestion on how
>> one might approach it.
>>
>>   2) The AAMI approach talks about agile development in a way that's
>>very
>> consistent with how most competent people would characterize agile
>> development.
>>
>>   3) I took a look at the Bruce Douglass report that Geoffrey Biggs
>> mentioned (Thanks, Geoffrey)
>>
>> 
>>http://www.ibm.com/developerworks/rational/library/agile-analysis-practic
>>es-safety-critical-development/
>>   I've worked with Bruce before, and I have a lot of respect for his
>>work.
>> However, I'd be willing to bet that most people who really understand
>>what
>> agile development is could easily argue that the Harmony method
>>discussed
>> by Bruce would not be considered agile. It's clearly iterative, but
>> iteration alone does not constitute true agility. *I* wouldn't consider
>> the Harmony method described by Bruce to be truly agile.
>>
>>
>>   -- steve
>>
>>
>>
>>    From: René Senden <rene.senden at gmail.com>
>> Date: Monday, September 2, 2013 8:32 AM
>> To: 'M Mencke' <menckem at gmail.com>
>> Cc: "systemsafety at techfak.uni-bielefeld.de" <
>> systemsafety at techfak.uni-bielefeld.de>
>>
>> Subject: Re: [SystemSafety] Agile methods
>>
>>    Myriam,****
>>
>> ** **
>>
>> You are right about my initial question being somewhat polarizing,
>>perhaps
>> because I tacitly assumed that anyone who¹d answer this question with
>> ³Yes², would also include some****
>>
>> of the corresponding experiences Š Regarding experience with
>> reconciliation of agile environments with typical/common/Š (software)
>> safety standards, I am very interested in the ****
>>
>> development processes and corresponding results/evidence/outputs, all
>>this
>> should be auditable. Is it (at all) possible to harmonize these very
>> different worlds, or would any such ****
>>
>> attempt result in compromising either? Transparency, auditability,
>> documentation, and safety specific processes such as analyses are
>>examples
>> that, to me, seem to be particularly ****
>>
>> difficult to address in an agile environment, let¹s take ³scrum² as an
>> example of a popular agile approach.****
>>
>> ** **
>>
>> Rene****
>>
>> ** **
>>
>> ** **
>>
>> *From:* systemsafety-bounces at lists.techfak.uni-bielefeld.de [
>> 
>>mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de<systemsafety-b
>>ounces at lists.techfak.uni-bielefeld.de>]
>> *On Behalf Of *M Mencke
>> *Sent:* maandag 2 september 2013 14:39
>> *To:* Jon Davies
>> *Cc:* systemsafety at techfak.uni-bielefeld.de
>> *Subject:* Re: [SystemSafety] Agile methods****
>>
>> ** **
>>
>> René,****
>>
>> I have come across a situation in the railway field where an
>> organization was required by the customer to comply with a safety
>>standard
>> which has specific requirements on the software development process, in
>> this case, EN 50128, where this was previously not required.  ****
>>
>> It seems to me that this type of situation typically arises in
>> industries/products where the SIL concept has traditionally not been
>> applied, or is being introduced. The customer creates a ³SIL
>>requirement²
>> as a type of marketing argument, where the SIL is understood by the
>> customer as applying to equipment rather than to a function (in order
>>to be
>> able to state ³This equipment is SIL X² in commercial scenarios).****
>>
>> Regarding your question, ³Have you encountered a situation, in
>>industrial
>> practice, in which an organization developing software following an
>>agile
>> methodology has to comply with a safety standard which has specific
>> requirements on the software development process?² ****
>>
>> This is a polar question, which can only be answered with a yes/no. ****
>>
>> If you are interested in practical experience, it might be useful to add
>> to this question which aspect(s) of such a situation you are interested
>>in.
>> For example, if the answer is ³yes², was the project actually a success?
>> Which measures/procedures were implemented in the company to approach
>>this
>> problem and reconcile the development process with such requirements?,
>>etc.
>> ****
>>
>> Depending on the type of information you are seeking, it may not be
>> possible to provide particular details regarding such a state of
>>affairs,
>> as this information may be company-specific.****
>>
>> Regards,****
>>
>> Myriam.****
>>
>> ** **
>>
>> 2013/9/2 Jon Davies <jdavies at theiet.org>****
>>
>> On 30 August 2013 18:02, René Senden <rene.senden at gmail.com> wrote:
>>> Dear all,
>>> ****
>>
>>> Do any of you have practical experience with reconciling established
>> agile
>>> software development with software safety requirements (e.g. IEC-61508
>>>or
>>> DO-178..) ?****
>>
>> Yes, and we usually end up throwing away the software developed using
>> "agile" methods, and starting again properly.
>>
>> I'm taking "agile software development" as meaning the development of
>> software using processes consistent with the agile manifesto:
>> http://agilemanifesto.org/ - to quote the relevant part:
>> "...we value... working software over comprehensive documentation"
>>
>> this is fundamentally in conflict with many of the things we know
>> about building high integrity software, and so "agile" methods are
>> fundamentally in conflict with developing software for safety critical
>> systems.
>>
>> There's plenty to learn from agile development methods that might be
>> useful in high integrity software development, but that's a whole
>> different discussion.  Every time we discuss agile development here,
>> we end up back at the need to use a development process that builds in
>> correctness - we can't test exhaustively, so we need a process that
>> builds integrity in.  Agile methods don't do this.
>>
>> Jon****
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE****
>>
>> ** **
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>>
>>
>
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>

-- 
Derek M. Jones                  tel: +44 (0) 1252 520 667
Knowledge Software Ltd          blog:shape-of-code.coding-guidelines.com
Software analysis               http://www.knosof.co.uk
_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE



More information about the systemsafety mailing list