3rd European-American Workshop  - Meetings & Breakout Session results


Reliability Working Group meeting September 9, 2002

Reliability committee working group meeting notes:

Meeting opened at 10am


Damir Marcocic– Croatia


Head of NDE lab

R&R focus within NDE

Working group V – European Joint Research – Focus mine detection (40 members ~15 active) human factors focus

Transfer standardization lessons learned to Mine detection


Christina Mueller – Head of NDE Reliability lab BAM, Focus on NDE and detection reliability since 1986

Ripi Singh – Program manager, Pratt Whitney

– Inspection reliability for USAF, 15 years reasearch. Structural integrity and reliability. Focus on maintainability and inspection program.

Lloyd Schaefer

Honeywell – Principal Engineer Global Repair Development Engineering. Focus on NDE systems development and reliability analysis in support of the life prediction effort for repairs of turbines

Sharon Vukelich – Head of efforts at USAF engine structural integrity program, related to NDE. 20 years experience, with 8 years GE engines. Author MIL-STD 1823 for reliability.


Peter Wilrich

Prof. Emer Frei Univ Berlin – Statistician – EE PhD – Tech Univ Hamburg

Measurement uncertainty. Chair ISO standards for interlab testing (esp precision) 5725 measurement reliability, also quantitative reliability. Looking at both demining and NDE. Capability 11843 ISO detection capability.

Perspectives on resources and actions within industrial and professional societies for statistical analysis of NDE and related processes:


Perspectives on use of statistical methods within industries and professional societies in Europe and America

Peter provided some perspectives on his experiences with the use and support of statistical methods within German and American industries:

ASQ stronger than Euro Quality society, with only ASQ possessing a statistical division.

6 Sigma just taking hold in EuroQuality and some industries, eg. Siemens


Peter shared his experiences with development/management of ISO interlab measurement reliability standards. A look ahead at an appropriate capability/detection standard targeted to demining activities, eg ISO 11843

Possible sources of support for creation of standards were discussed

DIN dropped interest in a measurement standard and the perception is that DIN will only get involved where there is strong industry backing.


The group discussed the need to determine currently what involvement/support is possible, and to establish a "pull" oriented feedback group, it was decided to create an action where AE reliability group members at this meeting would contact risk, lifing, and other reliability customer data groups to conduct a Voice of the Customer survey. This feedback would then be reflected on the web to motivate producers of NDE and detection reliability data, and also top-down industrial and government support for the efforts.


ASNT & DGZFP Reliability Committee status

Schaefer inquired as to the current level of effort and prognosis for the ASNT reliability committee effort – Sharon Vukelich and Christina Mueller provided status. There is a recognition of need to address it. 10 years ago, it was rejected as a "Standards" activity, and had perhaps objectionable implications for the membership and existing certification programs. The initial response to this posture of 10 years ago was that ASTM created a performance demonstration section related to its specific needs. Ripi Singh suggested that to avoid this problem that the primary influence be retained on testing system reliability, and muting the focus on individual inspector certification. While inherently problematic, such as in penetrant inspection, qualifying the inspector group capability as input to the design or lifing customer is a feasible work around.

All agreed that the focus should be on testing the "process" vs the individual.

Within ASNT board, both past president Doggert, and also Mike Turnbow supported the concept of a committee for NDE reliability.

To date ASNT committee is active, but have held only one meeting

With support of a neutrally located and focused NDE reliability web, knowledge of specimen and POD assessment resources and results will help germinate essentials which will lead towards understanding of global reliability by process, and identification of Critical to Quality elements which influence the results.

DGZfP had terminated its reliability committee, however, current president Jorge Volker will re-form this committee and champion the importance of acquiring and delivering this knowledge to customers.


Creation of white paper to capture Voice of the Customer for Reliability and Detection products

The reliability committee agreed that it should proceed to create a voice of the customer white paper which will guide further activities. Members will solicit input from


  • ASIP
  • ENSIP – Vince Spinnel – Structures, but with lower NDE focus. Spanel (vince)
  • Croatian register of shipping – (damir markovich),
  • Lloyd’s of London and additional insurance, risk management companies.
  • Mine action centers
  • Reliability doorway for demining through CEN-BT126 (working group) CWA7 (mine detectors)
    • Accepted modified performance demonstration requirement
  • ECNDT –


  1. The white paper categories – VOC, end users, will focus on:
  2. Petrochem
  3. Nuclear
  4. Commercial AF
  5. Military AF
  6. Commercial Engines
  7. Military engines
  8. Insurance/risk management
  9. NATO – Ward Rummel, Floyd Spencer, CNRC – David Forsyth



Potential for global assessment standards

The potential for a standard, globally, for reliability assessment design was discussed. Sharon Vukelich related past attempts which ended (AGARD) due to lack of agreement on experimental design. The PIE approach recommended by Ripi was then adopted as the best near term approach to allow practitioners to input the elements of their studies, and for the committee to evaluate what distills out of this process. There is clear value in measuring and understanding the variability of NDE process elements, and it is fundamentally valuable for reliability confidence when detection variables are in control, despite imperfect knowledge of absolute POD.

Structure of Partner Information Exchange:

The structure of the PIE will be:

PIE- Platform for information exchange

  • Publish results and their basis
  • Criteria and comments clearing section for each input
  • Additional emphasis on collecting and providing miss and false call distributions
  • Ultimately an internal standard will emerge from the compiled results
    • Adoption of the optimal procedure will distill on its own


Airport security reliability

To expand the mine detection portion of the committee area of interest, the issues related to weapons and explosives detection on persons and luggage/cargo will be examined. Lloyd Schaefer took an action to determine existing quality/reliability procedures for these processes and report back to the committee.

Development of failure and flaw distribution knowledge:

To understand better what specific targets of detection distributions should be, it is agreed that knowledge of especially failure related undetected, or detected and accepted flaws consist of. Sharon Vukelich indicated she will publish and distribute a database of all Aero rotor failure incidents through 2002 by the end of the year. Breakdown of the information will be per operator and engine/model.

Need for blind trials:

High human factors NDE and detection tasks have stronger needs for blind trials than majority automated systems. Examples, mine detection, airport security and FPI/MPI NDE tests. How to account for the known test factor, and how to effect blind trials and deliver the inspection planning information is of interest. To date, the use of backprojection techniques, such as the SIRS database enables the generation of "miss" distributions, Lloyd Schaefer reported. It was pointed out that when we have such information, it may not be longer necessary to conduct specific NDE reliability demonstrations. Lloyd Schaefer will write a report to the group on the nature and measurement of variables in the SIRS approach. It was agreed that blind trials are more readily possible for the mine and weapon detection reliability efforts, and the adequacy of these procedures will be evaluated when the existing quality procedures are obtained. Christina Mueller will provide procedure inputs for the demining procedures.

Actions and meeting closure:

Identified actions by the committee members will be carried out, and a status report compiled in 60 days. Meeting was adjourned at 2pm.


Session 1 - Creating and applying POD, ROC to risk and life management needs - Wednesday September 11th

Actual problems in creating and applying POD, ROC to risk and life management needs.

a) Test sample design

b) Finding the right POD/CL for the application

c) Optimizing assessment design

Main conclusion:

We need a catalogue of "risk based" POD curves

  Session 2 - The specifics of mine searching and the possibility to adopt NDE and NDE reliability methods


3rd European-American Workshop on NDE Reliability

Breakout Session 2

Sept 12. 2002, Berlin

The specifics of mine searching and the possibility to adopt NDE and NDE reliability





The goal of the Workshop was to:

  1. Accelerate the collaboration and information exchange within the humanitarian demining community (HDC), and across HD and NDT communities.
  2. Advance the state of the knowledge by recognizing and tackling the highest priority issues.
  3. Agree on the action plan / next steps to address the issues.


Add 1.

The WS included presentations by experts from Europe and North America addressing issues ranging from demining strategy and performance evaluation of modeling and field testing to actual field results of reliable mine clearance approaches.

The presenters shared their findings in field testing, demonstrated promising new methodologies, and provided a preview of new technology that will advance the reliability and speed of APMD and clearance and lower the cost per cleared area.


Add 2.

Additionally, participants had an opportunity to identify the highest priority issues in the demining activities area, and discuss the issues in a breakout session.

The two issues that surfaced during the session were:

The above issues result in a lowered ability to compare testing, modeling and field results and therefore xxxxxx the ability to predict the success for various new methods and technologies.

The issue results in the inability to xxxxxxxx done successful models, average existing knowledge and provide enough capacity, capability to meet the demining goals set by the international community.


Add 3.

In the breakout session the participants agreed on several act...... items / immediate next steps that would ......... in an environment to successfully tackle.

Both issues:


Main conclusion:

We should apply the NDE reliability formula for a deep understanding and improvement of the demining process: better educations (MAA) and performance demonstration for systems in the field.


Thursday, Sept. 12  Friday, Sept. 13

Session 3 - The Modular Model and Reliability Formula for NDE and Demining


Breakout Session Summary

The Modular Model and Reliability Formula for NDE and Demining


Philippe Benoist

Jay Fisher


The modular model as defined at a previous workshop is



At the beginning of the session, Christina Mueller summarized the modular model for the attendees.

There was a general consensus that the modular model has not been used mathematically. It was pointed out that the model was designed more as a way to understand factors that contribute to reliability, rather than to provide a mathematical approach to calculate inspection reliability.

Some participants thought that some quantitative use of the model might be possible. In particular, some claimed that IC and AP could be separated; others that HF could be separated from the other factors.

It was pointed out that different industries have developed different approaches to ensuring sufficient inspection reliability. In particular, the U.S. aerospace industry has embraced development of Probability of Detection (POD) curves, which essentially provide an overall measurement of inspection reliability. However, the U.S. nuclear industry has developed a performance demonstration that requires vendors to demonstrate a minimum level of detection and characterization on blind specimens, and only secondarily determines a quantitative measurement of inspection reliability. The European nuclear industry also has a qualification approach, but qualifies techniques and personnel separately, and uses demonstrations rather than blind tests. It was not clear that lessons from these industries could be easily applied to demining, which has a larger cost to less than 100% reliability and a large problem with false positives.

It was noted that existing published POD data are difficult to use: there is often not enough information to be able to judge how to apply existing data to a new situation. Christina Mueller suggested that a web site be created to collect data and standards on reliability. There was some discussion that it would be useful to develop common ways to describe terms that are used in describing reliability, so that reliability data could be easily understood and compared by people who did not generate the data.




Session 4 - How shall we combine calibration, reproducibility, and repeatability to POD/ROC investigations and the Modular Model

3rd European-American Workshop on NDE Reliability

Breakout Session 4

Sept 13. 2002, Berlin

How to combine calibration / R&R to POD/ROC investigations
and Modular model




Mike Anderson,

rely on performance demonstration.

what happens with calibration ? important is to detect.

until operator finds a crack and size it – no problem with calibration !

instruments are digitally controlled, self-calibration... algorithm

separate between manual and automatic calibration procedures

calibration ?!? (Bob)


standardization of procedure

POD can be used as a tool for process control

U. Ewert,

in RRT one have to describe how you collect data, formulae, put data, without list.

it is noticed that even in that case people do not obey instructions – because there was no computer program which could make multiple same calculations !

we need agreement on the tools

K. Osterloh,

one cannot compare crack with the mine :)

L. Schaefer,


bookkeeping of former results to trace information about calibrated thresholds

H. Stephens,

platform for comparison of results, design of standards with "typical" flaws

short term need:

couple of choices how to collect data

D. Markucic,

with each number extraction or even a single curve (POD) we loose quality of data

calibration "instruction" is a part of data description

Ripi Singh,

data are not portable, case stories !

information of knowledge

H. Stephens,

Lack of education (knowledge) for people outside NDE community with which we work/communicate. Need to move beyond POD = badness (Lloyd). Articulate the message to our NDT societies about this topics.

Main conclusion:

Traceability of results; Standardization of procedure; POD for process control (6 sigma)



  Session 5 - Summary & Consensus

Summary & Consensus.

Breakout Session 5

(R.Singh, M.Anderson, C.Mueller)

(A.Dean, K.Osterloh, J.O.Robertz, C.Mueller)


Main conclusion:

We need to quantify the risk in NDE and demining !




fuselage_ec_results.htm ] resume.htm ] LASchaeferCV.htm ] Birdmail.htm ]