Automatic Evaluation of Robstness and Degradation in 在鲁棒性和降解的自动评价.pptVIP

Automatic Evaluation of Robustness and Degradation in Tagging and Parsing Johnny Bigert, Ola Knutsson, Jonas Sj?bergh Royal Institute of Technology, Stockholm, Sweden Contact: johnny@kth.se Problem NLP systems are often faced with noisy and ill-formed input: How do we reliably evaluate the performance of NLP systems? Which methods of tagging and parsing are robust? Problem The performance of a NLP system is sensitive to noisy and ill-formed input Manual evaluations of robustness is tedious and time-consuming Manual evaluation is difficult to compare and reproduce Resources with noisy data is rare Outline Introduce artificial spelling errors using software (Missplel) Increasing error levels will affect the NLP system performance Evaluation of degradation of tagging and parsing performance (AutoEval) Introducing spelling errors Missplel (Bigert et al) Generic tool to introduce human-like spelling errors Highly configurable Language and tag set independent Freeware, open source http://www.nada.kth.se/theory/humanlang/tools.html Introducing spelling errors Start with correct text (Swedish, the SUC corpus, Ejerhed et al) Introduce errors in, say, 10% of the words Spelling errors resulting in non-existing words only No change in parse tree Introducing spelling errors 10 misspelled texts for each error level Eliminate the influence of chance Six error levels:0%, 1%, 2%, 5%, 10%, 20% 15 000 words with parse info Missplel example Letters NN2 would VM0 be VBI welcome AJ0-NN1 Litters NN2 damerau/wordexist-notagchange would VM0 ok bee NN1 sound/wordexist-tagchange welcmoe ERR damerau/nowordexist-tagchange Tagging The texts were tagged using HMM tagger (TnT, Brants) Brill tagger (fnTBL, Ngai Florian) Baseline tagger (unigram) Parsing The tagged texts were parsed using GTA parser (Knutsson et al) Baseline parser (unigram, CoNLL) GTA - Granska text analyzer Rule-based Hand-crafted rules Context-free formalism