Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity.pdfVIP

Journal of Clinical Epidemiology 53 (2000) 499–503 0895-4356/00/$ – see front matter ? 2000 Elsevier Science Inc. All rights reserved. PII: S0895-4356(99)00174-2 Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity Fred K. Hoehler* Data Management Center, 210 South Batavia Street, Orange, CA 92868, USA Received 14 July 1999; received in revised form 29 July 1999; accepted 23 August 1999 Abstract Paradoxical effects of bias and prevalence on the kappa coefficient are examined using the concepts of sensitivity and specificity. Re- sults that appear paradoxical when viewed as a 2 3 2 table of frequencies do not appear paradoxical when viewed as a pair of sensitivity and specificity measures where each observer is treated as a predictor of the other observer. An adjusted kappa value can be obtained from these sensitivity/specificity measures but simulation studies indicate that it would result in substantial overestimation of reliability when bias or prevalence effects are observed. It is suggested that investigators concentrate on obtaining populations with trait prevalence near 50% rather than searching for statistical indices to rescue or excuse inefficient experiments. ? 2000 Elsevier Science Inc. All rights reserved. Keywords: Kappa; Sensitivity; Specificity; ROC; Bias; Prevalence 1. Introduction The kappa coefficient, first described by Cohen in 1960 [1], is commonly used to assess agreement between two or more observers. In its simplest form, it is used to analyze the case where two observers classify a number of cases ac- cording to whether some finding (e.g., a tumor or other dis- ease process) is present or absent. The results may be ar- ranged in the four cells of a 2 3 2 table as shown below. Cell a indicates the proportion of cases where both observ- ers agree that the finding is present. Cell b indicates the pro- portion of cases where observer #2 claims that the finding is pres