Are Current Prognostic Performance Evaluation Practices Sufficient and Meaningful?

Shankar Sankararaman, Abhinav Saxena, and Kai Goebel
Submission Type: 
Full Paper
phmc_14_082.pdf448.01 KBSeptember 26, 2014 - 9:16am

This paper investigates the shortcomings of performance evaluation for prognostic algorithms, particularly in the presence of uncertainty. To that end, the various elements of a prognostic algorithm (present health state estimation, future load condition, degradation model, and damage threshold) and their effects on prognostics are examined. Each of these elements contribute to overall prediction performance and therefore it is important to distinguish between (1) assessment of the correctness of information regarding these quantities, and (2) the assessment of correctness of the prognostic algorithm. The need for proper accounting for uncertainty in the various associated elements is discussed. Next, the shortcomings of traditional comparisons between ground truth and algorithm prediction is discussed. Several scenarios are pointed out where misleading interpretations about evaluation outcomes are possible. In order to address these shortcomings an ``informed evaluation'' methodology is being proposed, where the algorithm is informed with future loading/operating conditions before comparing against ground truth. Additionally, the importance of estimating the accuracy of aggregating the different sources of uncertainty using rigorous mathematical procedures is also emphasized. While this discussion does not target developing new metrics, it highlights key criteria for an accurate performance evaluation process under uncertainty and proposes new measures to accomplish this goal.

Publication Year: 
Publication Volume: 
Publication Control Number: 
Page Count: 
Submission Keywords: 
performance evaluation
Submission Topic Areas: 
Uncertainty Quantification and Management in PHM
Verification and validation
Submitted by: 

follow us

PHM Society on Facebook Follow PHM Society on Twitter PHM Society on LinkedIn PHM Society RSS News Feed