3rd European-American Workshop - Meetings & Breakout Session results
Reliability Working Group meeting September 9, 2002
Meeting opened at 10am
Damir Marcocic Croatia
Head of NDE lab
R&R focus within NDE
Working group V European Joint Research Focus mine detection (40 members ~15 active) human factors focus
Transfer standardization lessons learned to Mine detection
Christina Mueller Head of NDE Reliability lab BAM, Focus on NDE and detection reliability since 1986
Ripi Singh Program manager, Pratt Whitney
Inspection reliability for USAF, 15 years reasearch. Structural integrity and reliability. Focus on maintainability and inspection program.
Honeywell Principal Engineer Global Repair Development Engineering. Focus on NDE systems development and reliability analysis in support of the life prediction effort for repairs of turbines
Sharon Vukelich Head of efforts at USAF engine structural integrity program, related to NDE. 20 years experience, with 8 years GE engines. Author MIL-STD 1823 for reliability.
Prof. Emer Frei Univ Berlin Statistician EE PhD Tech Univ Hamburg
Measurement uncertainty. Chair ISO standards for interlab testing (esp precision) 5725 measurement reliability, also quantitative reliability. Looking at both demining and NDE. Capability 11843 ISO detection capability.
Perspectives on resources and actions within industrial and professional societies for statistical analysis of NDE and related processes:
Perspectives on use of statistical methods within industries and professional societies in Europe and America
Peter provided some perspectives on his experiences with the use and support of statistical methods within German and American industries:
ASQ stronger than Euro Quality society, with only ASQ possessing a statistical division.
6 Sigma just taking hold in EuroQuality and some industries, eg. Siemens
Peter shared his experiences with development/management of ISO interlab measurement reliability standards. A look ahead at an appropriate capability/detection standard targeted to demining activities, eg ISO 11843
Possible sources of support for creation of standards were discussed
DIN dropped interest in a measurement standard and the perception is that DIN will only get involved where there is strong industry backing.
The group discussed the need to determine currently what involvement/support is possible, and to establish a "pull" oriented feedback group, it was decided to create an action where AE reliability group members at this meeting would contact risk, lifing, and other reliability customer data groups to conduct a Voice of the Customer survey. This feedback would then be reflected on the web to motivate producers of NDE and detection reliability data, and also top-down industrial and government support for the efforts.
ASNT & DGZFP Reliability Committee status
Schaefer inquired as to the current level of effort and prognosis for the ASNT reliability committee effort Sharon Vukelich and Christina Mueller provided status. There is a recognition of need to address it. 10 years ago, it was rejected as a "Standards" activity, and had perhaps objectionable implications for the membership and existing certification programs. The initial response to this posture of 10 years ago was that ASTM created a performance demonstration section related to its specific needs. Ripi Singh suggested that to avoid this problem that the primary influence be retained on testing system reliability, and muting the focus on individual inspector certification. While inherently problematic, such as in penetrant inspection, qualifying the inspector group capability as input to the design or lifing customer is a feasible work around.
All agreed that the focus should be on testing the "process" vs the individual.
Within ASNT board, both past president Doggert, and also Mike Turnbow supported the concept of a committee for NDE reliability.
To date ASNT committee is active, but have held only one meeting
With support of a neutrally located and focused NDE reliability web, knowledge of specimen and POD assessment resources and results will help germinate essentials which will lead towards understanding of global reliability by process, and identification of Critical to Quality elements which influence the results.
DGZfP had terminated its reliability committee, however, current president Jorge Volker will re-form this committee and champion the importance of acquiring and delivering this knowledge to customers.
Creation of white paper to capture Voice of the Customer for Reliability and Detection products
The reliability committee agreed that it should proceed to create a voice of the customer white paper which will guide further activities. Members will solicit input from
Potential for global assessment standards
The potential for a standard, globally, for reliability assessment design was discussed. Sharon Vukelich related past attempts which ended (AGARD) due to lack of agreement on experimental design. The PIE approach recommended by Ripi was then adopted as the best near term approach to allow practitioners to input the elements of their studies, and for the committee to evaluate what distills out of this process. There is clear value in measuring and understanding the variability of NDE process elements, and it is fundamentally valuable for reliability confidence when detection variables are in control, despite imperfect knowledge of absolute POD.
Structure of Partner Information Exchange:
The structure of the PIE will be:
PIE- Platform for information exchange
Airport security reliability
To expand the mine detection portion of the committee area of interest, the issues related to weapons and explosives detection on persons and luggage/cargo will be examined. Lloyd Schaefer took an action to determine existing quality/reliability procedures for these processes and report back to the committee.
Development of failure and flaw distribution knowledge:
To understand better what specific targets of detection distributions should be, it is agreed that knowledge of especially failure related undetected, or detected and accepted flaws consist of. Sharon Vukelich indicated she will publish and distribute a database of all Aero rotor failure incidents through 2002 by the end of the year. Breakdown of the information will be per operator and engine/model.
Need for blind trials:
High human factors NDE and detection tasks have stronger needs for blind trials than majority automated systems. Examples, mine detection, airport security and FPI/MPI NDE tests. How to account for the known test factor, and how to effect blind trials and deliver the inspection planning information is of interest. To date, the use of backprojection techniques, such as the SIRS database enables the generation of "miss" distributions, Lloyd Schaefer reported. It was pointed out that when we have such information, it may not be longer necessary to conduct specific NDE reliability demonstrations. Lloyd Schaefer will write a report to the group on the nature and measurement of variables in the SIRS approach. It was agreed that blind trials are more readily possible for the mine and weapon detection reliability efforts, and the adequacy of these procedures will be evaluated when the existing quality procedures are obtained. Christina Mueller will provide procedure inputs for the demining procedures.
Actions and meeting closure:
Identified actions by the committee members will be carried out, and a status report compiled in 60 days. Meeting was adjourned at 2pm.
Session 1 - Creating and applying POD, ROC to risk and life
management needs - Wednesday September 11th
Actual problems in creating and applying POD, ROC to risk and life management needs.
a) Test sample design
b) Finding the right POD/CL for the application
c) Optimizing assessment design
We need a catalogue of "risk based" POD curves
Session 2 - The specifics of mine searching and the possibility to adopt
NDE and NDE reliability methods|
3rd European-American Workshop on NDE Reliability
Breakout Session 2
Sept 12. 2002, Berlin
The specifics of mine searching and the possibility to adopt NDE and NDE reliability
The goal of the Workshop was to:
The WS included presentations by experts from Europe and North America addressing issues ranging from demining strategy and performance evaluation of modeling and field testing to actual field results of reliable mine clearance approaches.
The presenters shared their findings in field testing, demonstrated promising new methodologies, and provided a preview of new technology that will advance the reliability and speed of APMD and clearance and lower the cost per cleared area.
Additionally, participants had an opportunity to identify the highest priority issues in the demining activities area, and discuss the issues in a breakout session.
The two issues that surfaced during the session were:
The above issues result in a lowered ability to compare testing, modeling and field results and therefore xxxxxx the ability to predict the success for various new methods and technologies.
The issue results in the inability to xxxxxxxx done successful models, average existing knowledge and provide enough capacity, capability to meet the demining goals set by the international community.
In the breakout session the participants agreed on several act...... items / immediate next steps that would ......... in an environment to successfully tackle.
We should apply the NDE reliability formula for a deep understanding and improvement of the demining process: better educations (MAA) and performance demonstration for systems in the field.
Thursday, Sept. 12 Friday, Sept. 13
Breakout Session Summary
The Modular Model and Reliability Formula for NDE and Demining
The modular model as defined at a previous workshop is
At the beginning of the session, Christina Mueller summarized the modular model for the attendees.
There was a general consensus that the modular model has not been used mathematically. It was pointed out that the model was designed more as a way to understand factors that contribute to reliability, rather than to provide a mathematical approach to calculate inspection reliability.
Some participants thought that some quantitative use of the model might be possible. In particular, some claimed that IC and AP could be separated; others that HF could be separated from the other factors.
It was pointed out that different industries have developed different approaches to ensuring sufficient inspection reliability. In particular, the U.S. aerospace industry has embraced development of Probability of Detection (POD) curves, which essentially provide an overall measurement of inspection reliability. However, the U.S. nuclear industry has developed a performance demonstration that requires vendors to demonstrate a minimum level of detection and characterization on blind specimens, and only secondarily determines a quantitative measurement of inspection reliability. The European nuclear industry also has a qualification approach, but qualifies techniques and personnel separately, and uses demonstrations rather than blind tests. It was not clear that lessons from these industries could be easily applied to demining, which has a larger cost to less than 100% reliability and a large problem with false positives.
It was noted that existing published POD data are difficult to use: there is often not enough information to be able to judge how to apply existing data to a new situation. Christina Mueller suggested that a web site be created to collect data and standards on reliability. There was some discussion that it would be useful to develop common ways to describe terms that are used in describing reliability, so that reliability data could be easily understood and compared by people who did not generate the data.
Session 4 - How shall we combine calibration, reproducibility, and repeatability to POD/ROC investigations and the Modular Model
3rd European-American Workshop on NDE Reliability
Breakout Session 4
Sept 13. 2002, Berlin
How to combine calibration / R&R to POD/ROC
and Modular model
rely on performance demonstration.
what happens with calibration ? important is to detect.
until operator finds a crack and size it no problem with calibration !
instruments are digitally controlled, self-calibration... algorithm
separate between manual and automatic calibration procedures
calibration ?!? (Bob)
standardization of procedure
POD can be used as a tool for process control
in RRT one have to describe how you collect data, formulae, put data, without list.
it is noticed that even in that case people do not obey instructions because there was no computer program which could make multiple same calculations !
we need agreement on the tools
one cannot compare crack with the mine :)
bookkeeping of former results to trace information about calibrated thresholds
platform for comparison of results, design of standards with "typical" flaws
short term need:
couple of choices how to collect data
with each number extraction or even a single curve (POD) we loose quality of data
calibration "instruction" is a part of data description
data are not portable, case stories !
information of knowledge
Lack of education (knowledge) for people outside NDE community with which we work/communicate. Need to move beyond POD = badness (Lloyd). Articulate the message to our NDT societies about this topics.
Traceability of results; Standardization of procedure; POD for process control (6 sigma)
Summary & Consensus.
Breakout Session 5
(R.Singh, M.Anderson, C.Mueller)
(A.Dean, K.Osterloh, J.O.Robertz, C.Mueller)
We need to quantify the risk in NDE and demining !
CLICK BELOW FOR BREAKOUT SESSIONS CONCLUSIONS