how to get the most out of your curation effort如何充分利用你的管理工作.pdfVIP

How to Get the Most out of Your Curation Effort 1,2,3,4 5 6 Andrey Rzhetsky *, Hagit Shatkay , W. John Wilbur 1 Department of Medicine, University of Chicago, Chicago, Illinois, United States of America, 2 Department of Human Genetics, University of Chicago, Chicago, Illinois, United States of America, 3 Computation Institute, University of Chicago, Chicago, Illinois, United States of America, 4 Institute for Genomics and Systems Biology, University of Chicago, Chicago, Illinois, United States of America, 5 The Computational Biology and Machine Learning Lab, School of Computing, Queen’s University, Kingston, Ontario, Canada, 6 National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, Maryland, United States of America Abstract Large-scale annotation efforts typically involve several experts who may disagree with each other. We propose an approach for modeling disagreements among experts that allows providing each annotation with a confidence value (i.e., the posterior probability that it is correct). Our approach allows computing certainty-level for individual annotations, given annotator-specific parameters estimated from data. We developed two probabilistic models for performing this analysis, compared these models using computer simulation, and tested each model’s actual performance, based on a large data set generated by human annotators specifically for this study. We show that even in the worst-case scenario, when all annotators disagree, our approach allows us to significantly increase the probability of choosing the correct annotation. Along with this publication we make publicly available a corpus of 10,000 sentences annotated according to several cardinal