[SystemSafety] Bossavit's Leprechauns book

Sat Dec 8 04:15:54 CET 2018

Sounds like they were aiming for a randomized trial? 

Matthew Squair

MIEAust, CPEng
Mob: +61 488770655
Email; Mattsquair at gmail.com
Web: http://criticaluncertainties.com

> On 7 Dec 2018, at 6:47 pm, Nick Tudor <njt at tudorassoc.com> wrote:
> 
> You’re right, conducting trials is not easy, unless you have partial funding to conduct such benchmarking trials. We were part of an industrial trial last year using one of our embryonic FM tools vs a baseline of people only vs an incumbent. The trial seeded 48 errors into a real exemplar; I understand that these were typical and atypical errors which were ‘designed’ by both a University and the industrial participants. In other words, we had nothing to do with designing the trial; we merely provided our embryonic tool into the trial. The 3 strands of the trial were conducted by 3 different people on 6 parts of an ASIL D software design so that there was no pollution of knowledge. One of the 6 parts was repeated in another company; ie completely independently. The experiment was design by a specialist metrics company and t’was they that policed and produced the results. The experiment was to measure the time taken to satisfy the review process between requirements and design in Simulink. 
> 
> Not only did our tool find all the errors, we also found an extra one. Against the benchmark savings of 60-80% were found and against the incumbent 50-60%. The repeated part gave almost exactly the same results as the other; within a few %  for each of the 3 strands. There is a paper; if you’re interested contact me at njt at drisq.com. The embryonic tools are now almost at production level (Jan’19) and we will be talking about other uses at SSS’18. 
> 
> 
>> On Thu, 6 Dec 2018 at 23:10, Olwen Morgan <olwen at phaedsys.com> wrote:
>> 
>> While, ideally, we should base our engineering on what we can prove to 
>> be true, the argument that things are not true because there is no hard 
>> evidence for them is scientific dilettantism. There is ample reason to 
>> question the validity of various claims in software engineering not 
>> because they are not true but because there are many reasons why the 
>> quality of the evidence may be sufficiently poor that they are not 
>> proven. Not least among such reasons is the technical difficulty and 
>> cost of conducting properly controlled trials.
>> 
>> Not true and not proven are not synonymous - as we all learned from the 
>> proof of Fermat's Last Theorem by Andrew Wiles.
>> 
>> If you want to be a hard-nosed empiricist, you need to be rigorous about 
>> the design of the experiments by which you test hypotheses. If you want 
>> to question the validity of published results, that's fine ... but if 
>> you do, you need to look at all the circumstances in which a study has 
>> been carried out and document any flaws that you find in the methods 
>> used. Claims of no proof must be supported by hard evidence of flawed 
>> investigation.
>> 
>> How many reported studies does Bossavit examine with due scientific rigour?
>> 
>> Find that out and you'll know whether he's hoisted himself by his own 
>> petard.
>> 
>> 
>> Olwen
>> 
>> 
>> On 06/12/2018 22:00, Les Chambers wrote:
>> > Derek
>> > Thanks but Shock horror.  Sounds like he is devaluing some of the fundamental truths of SE.
>> > For example the ‘rising cost’ concept is the basis for spending millions on V&V. Righteous
>> > dollars.
>> > Have we a tech Trump amongst us pedaling fake news with honeyed words?
>> > Seriously though, this type of BS does not advance the SE canon.
>> > Bin it!
>> > Les
>> >
>> >> Les,
>> >>
>> >>> Is Bossavit supporting or rejecting the legends. Having read his book, what does he want
>> > you to
>> >>> do with the information.
>> >>   From the open chapter:
>> >>
>> >> "The software profession has a problem, widely recognized but
>> >> which nobody seems willing to do much about. You can think of
>> >> this problem as a variant of the well known Â¿telephone gameÂ¿,
>> >> where some trivial rumor is repeated from one person to the next
>> >> until it has become distorted beyond recognition and blown up
>> >> out of all proportion.
>> >>
>> >> Unfortunately, the objects of this telephone game are generally
>> >> considered cornerstone truths of the discipline, to the point that
>> >> their acceptance now hinders further progress.
>> >>
>> >> It is not that these claims are outlandish in themselves; they
>> >> started as somewhat reasonable hypotheses. The problem is that
>> >> they have become entrenched as Â¿factÂ¿ supposedly supported
>> >> by Â¿researchÂ¿, and attained this elevated status in spite of being
>> >> merely anecdotal.
>> >>
>> >> How we got there
>> >>
>> >> One of the ways that anecdote persists is by dressing itself up
>> >> in the garments of proper scholarship. Suppose you come across
>> >> the following claim for the first time:..."
>> >>
>> >>> Les
>> >>>
>> >>>> Les,
>> >>>>
>> >>>>> PBL
>> >>>>> Please explain. Is Bossavit denigrating these concepts by calling them memes, myths or
>> >>> legends? Is he
>> >>>> Peter has jumbled together commentary on Bossavit's book, commentary
>> >>>> on what people on this list have been talking about and his own views.
>> >>>>
>> >>>> Bossavit writes in a very readable style and it is clear what
>> >>>> his discovered (or rather not discovered, i.e., data backing up
>> >>>> the claims investigated).
>> >>>>
>> >>>> -- 
>> >>>> Derek M. Jones           Software analysis
>> >>>> tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com
>> >>>> _______________________________________________
>> >>>> The System Safety Mailing List
>> >>>> systemsafety at TechFak.Uni-Bielefeld.DE
>> >>>> Manage your subscription: https://lists.techfak.uni-
>> > bielefeld.de/mailman/listinfo/systemsafety
>> >>>
>> >>>
>> >>> --
>> >>> Les Chambers
>> >>> les at chambers.com.au
>> >>> +61 (0)412 648 992
>> >>>
>> >>>
>> >>>
>> >> -- 
>> >> Derek M. Jones           Software analysis
>> >> tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com
>> >
>> >
>> > --
>> > Les Chambers
>> > les at chambers.com.au
>> > +61 (0)412 648 992
>> >
>> >
>> > _______________________________________________
>> > The System Safety Mailing List
>> > systemsafety at TechFak.Uni-Bielefeld.DE
>> > Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
>> >
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety at TechFak.Uni-Bielefeld.DE
>> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
> -- 
> Nick Tudor
> Tudor Associates Ltd
> Mobile: +44(0)7412 074654
> www.tudorassoc.com
> 
> 
> 77 Barnards Green Road
> Malvern
> Worcestershire
> WR14 3LR
> Company No. 07642673
> VAT No:116495996
> 
> www.aeronautique-associates.com
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20181208/c52d20eb/attachment.html>