[SystemSafety] Agile methods

Mon Sep 2 19:07:59 CEST 2013

Does the AAMI report do any scientific analysis of the errors or recalls
involved in the medical device software produced by Agile methods? Or is it
simple "we used it"?

Given the sad state of medical device software, it does not seem that
simply using Agile methods proves anything. Homa Alenzadeh, a grad student
at UIUC, has looked at medical device recalls. The following is from an
abstract she wrote for a talk she gave: "Medical device incidents are one
of the leading causes of serious injury and death in the United States.
Between years 2006 and 2011, 5,294 recalls and approximately 1.2 million
adverse events of medical devices were reported to the US Food and Drug
Administration (FDA). Almost 23 percent of these recalls were due to
computer-related failures, of which approximately 94 percent presented
medium to high risk of severe health consequences (such as serious injury
or death) to patients.   '

If my arithmetic is correct (1,200,000*.23 and then divide by 5 then
multiply by .94), it appears from Homa's data that there are 51,888
reported potentially serious incidents to the FDA regarding software each
year. That is 144 every day. And many incidents are not reported (the
reporting rules are pretty sketchy) and that is only in the U.S.

The simple fact that mdedical device manufacturers have used Agile methods
does not prove anything about their efficacy for safety-critical systems
given the sad state of safety in that industry.

Nncy

On Mon, Sep 2, 2013 at 12:40 PM, Steve Tockey <Steve.Tockey at construx.com>wrote:

>  All,
> I have personal experience using Agile, and personal experience doing
> safety-critical development (DO-178B), but not personal experience doing
> agile development for a safety critical system. Instead of being polarizing
> and claiming "it can't be done", I'll simply say that I believe that it is
> possible to do it—it could just end up being very costly to do so. The
> worst case would be developing the code for the system in an truly agile
> way, and then looping back and re-developing it with the necessary safety
> analysis, etc. in place using more traditional processes.
>
>  All that said, I have a report titled, "AAMI
> TIR45: 2012 Technical Information Report      Guidance on the use of AGILE
> practices in the development of medical device software". AAMI is the
> "Association for the Advancment of Medical Instrumentation" and they're
> based at
> 4301 N. Fairfax Drive, Suite 301
> Arlington, VA   USA    22203-1633
> www.aami.org
>
>  The report abstract states, "Over the past several years, AGILE software
> development has become an accepted method for developing software products.
> There have been questions from both manufacturers and regulators as to
> whether (or which) AGILE practices are appropriate for developing medical
> device software. Enough medical device manufacturers have implemented AGILE
> practices in their software development so that answers to these questions
> can be documented. Having clear guidance of which practices have been found
> to be appropriate will be very useful for all developers of medical device
> software. This TIR will provide recommendations for complying with
> international standards and U.S. Food and Drug Administration (FDA)
> guidance documents when using AGILE practices to develop medical device
> software."
>
>  A preview copy is at:
>
> http://marketplace.aami.org/eseries/scriptcontent/docs/Preview%20Files/TIR45_1208_preview.pdf
>
>  I have a licensed copy, but it's copyrighted material and can't be
> shared. If you're really interested in the content, you'll have to get a
> copy on your own (sorry).
>
>  Having said that, three comments about all this:
>
>  1) I don't know what sort of status AAMI has with respect to FDA policy
> on medical device software development. In other words, I don't know if the
> FDA has actually agreed to this, or it's simply AAMI's suggestion on how
> one might approach it.
>
>  2) The AAMI approach talks about agile development in a way that's very
> consistent with how most competent people would characterize agile
> development.
>
>  3) I took a look at the Bruce Douglass report that Geoffrey Biggs
> mentioned (Thanks, Geoffrey)
>
> http://www.ibm.com/developerworks/rational/library/agile-analysis-practices-safety-critical-development/
>  I've worked with Bruce before, and I have a lot of respect for his work.
> However, I'd be willing to bet that most people who really understand what
> agile development is could easily argue that the Harmony method discussed
> by Bruce would not be considered agile. It's clearly iterative, but
> iteration alone does not constitute true agility. *I* wouldn't consider
> the Harmony method described by Bruce to be truly agile.
>
>
>  -- steve
>
>
>
>   From: René Senden <rene.senden at gmail.com>
> Date: Monday, September 2, 2013 8:32 AM
> To: 'M Mencke' <menckem at gmail.com>
> Cc: "systemsafety at techfak.uni-bielefeld.de" <
> systemsafety at techfak.uni-bielefeld.de>
>
> Subject: Re: [SystemSafety] Agile methods
>
>   Myriam,****
>
> ** **
>
> You are right about my initial question being somewhat polarizing, perhaps
> because I tacitly assumed that anyone who’d answer this question with
> “Yes”, would also include some****
>
> of the corresponding experiences … Regarding experience with
> reconciliation of agile environments with typical/common/… (software)
> safety standards, I am very interested in the ****
>
> development processes and corresponding results/evidence/outputs, all this
> should be auditable. Is it (at all) possible to harmonize these very
> different worlds, or would any such ****
>
> attempt result in compromising either? Transparency, auditability,
> documentation, and safety specific processes such as analyses are examples
> that, to me, seem to be particularly ****
>
> difficult to address in an agile environment, let’s take “scrum” as an
> example of a popular agile approach.****
>
> ** **
>
> Rene****
>
> ** **
>
> ** **
>
> *From:* systemsafety-bounces at lists.techfak.uni-bielefeld.de [
> mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de<systemsafety-bounces at lists.techfak.uni-bielefeld.de>]
> *On Behalf Of *M Mencke
> *Sent:* maandag 2 september 2013 14:39
> *To:* Jon Davies
> *Cc:* systemsafety at techfak.uni-bielefeld.de
> *Subject:* Re: [SystemSafety] Agile methods****
>
> ** **
>
> René,****
>
> I have come across a situation in the railway field where an
> organization was required by the customer to comply with a safety standard
> which has specific requirements on the software development process, in
> this case, EN 50128, where this was previously not required.  ****
>
> It seems to me that this type of situation typically arises in
> industries/products where the SIL concept has traditionally not been
> applied, or is being introduced. The customer creates a “SIL requirement”
> as a type of marketing argument, where the SIL is understood by the
> customer as applying to equipment rather than to a function (in order to be
> able to state “This equipment is SIL X” in commercial scenarios).****
>
> Regarding your question, “Have you encountered a situation, in industrial
> practice, in which an organization developing software following an agile
> methodology has to comply with a safety standard which has specific
> requirements on the software development process?” ****
>
> This is a polar question, which can only be answered with a yes/no. ****
>
> If you are interested in practical experience, it might be useful to add
> to this question which aspect(s) of such a situation you are interested in.
> For example, if the answer is “yes”, was the project actually a success?
> Which measures/procedures were implemented in the company to approach this
> problem and reconcile the development process with such requirements?, etc.
> ****
>
> Depending on the type of information you are seeking, it may not be
> possible to provide particular details regarding such a state of affairs,
> as this information may be company-specific.****
>
> Regards,****
>
> Myriam.****
>
> ** **
>
> 2013/9/2 Jon Davies <jdavies at theiet.org>****
>
> On 30 August 2013 18:02, René Senden <rene.senden at gmail.com> wrote:
> > Dear all,
> >****
>
> > Do any of you have practical experience with reconciling established
> agile
> > software development with software safety requirements (e.g. IEC-61508 or
> > DO-178..) ?****
>
> Yes, and we usually end up throwing away the software developed using
> "agile" methods, and starting again properly.
>
> I'm taking "agile software development" as meaning the development of
> software using processes consistent with the agile manifesto:
> http://agilemanifesto.org/ - to quote the relevant part:
> "...we value... working software over comprehensive documentation"
>
> this is fundamentally in conflict with many of the things we know
> about building high integrity software, and so "agile" methods are
> fundamentally in conflict with developing software for safety critical
> systems.
>
> There's plenty to learn from agile development methods that might be
> useful in high integrity software development, but that's a whole
> different discussion.  Every time we discuss agile development here,
> we end up back at the need to use a development process that builds in
> correctness - we can't test exhaustively, so we need a process that
> builds integrity in.  Agile methods don't do this.
>
> Jon****
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE****
>
> ** **
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>

-- 
Prof. Nancy Leveson
Aeronautics and Astronautics and Engineering Systems
MIT, Room 33-334
77 Massachusetts Ave.
Cambridge, MA 02142

Telephone: 617-258-0505
Email: leveson at mit.edu
URL: http://sunnyday.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20130902/d786241a/attachment-0001.html>