Attribute Agreement Assessment

Gepostet von am Dez 3, 2020 in Allgemein | Keine Kommentare

An attribute analysis was developed to simultaneously assess the effects of repeatability and reproducibility on accuracy. It allows the analyst to review the responses of several reviewers if they look at multiple scenarios multiple times. It establishes statistics that assess the ability of evaluators to agree with themselves (repeatability), with each other (reproducibility) and with a master or correct value (overall accuracy) known for each characteristic – over and over again. The attribute analysis study can be implemented in the same way as a regular study of Gauge R-R. A number of parts are selected in the process and evaluated by two or more operators. The study will determine the extent to which operators are consistent in their own assessments and to what extent operators are consistent. While it is possible to set a standard for the evaluation of each part, each operator`s performance can also be compared to the standard. Modern statistical software such as Minitab can be used to collect study data and perform analysis. The output and kappa graphics can be used to verify the effectiveness and accuracy of operators in conducting their evaluations. In this example, a repeatability assessment is used to illustrate the idea, and it also applies to reproducibility. The fact is that many samples are needed to detect differences in an analysis of the attribute, and if the number of samples is doubled from 50 to 100, the test does not become much more sensitive.

Of course, the difference that needs to be identified depends on the situation and the level of risk that the analyst is prepared to bear in the decision, but the reality is that in 50 scenarios, it is difficult for an analyst to think that there is a statistical difference in the reproducibility of two examiners with match rates of 96 percent and 86 percent. With 100 scenarios, the analyst will not be able to see any difference between 96% and 88%. Analytically, this technique is a wonderful idea. But in practice, the technique can be difficult to execute judiciously. First, there is always the question of sample size. For attribute data, relatively large samples are required to be able to calculate percentages with relatively low confidence intervals. If an expert looks at 50 different error scenarios – twice – and the match rate is 96 percent (48 votes vs. 50), the 95 percent confidence interval ranges from 86.29% to 99.51 percent. It is a fairly large margin of error, especially in terms of the challenge of choosing the scenarios, checking them in depth, making sure the value of the master is assigned, and then convincing the examiner to do the job – twice.