Improved Source-Channel Models for Chinese Word Segmentation 1.pdfVIP

Improved Source-Channel Models for Chinese Word Segmentation1 Jianfeng Gao, Mu Li and Chang-Ning Huang Microsoft Research, Asia Beijing 100080, China {jfgao, t-muli, cnhuang}@ 1 We would like to thank Ashley Chang, Jian-Yun Nie, Andi Wu and Ming Zhou for many useful discussions, and for comments on earlier versions of this paper. We would also like to thank Xiaoshan Fang, Jianfeng Li, Wenfeng Yang and Xiaodan Zhu for their help with evaluating our system. Abstract This paper presents a Chinese word segmen- tation system that uses improved source- channel models of Chinese sentence genera- tion. Chinese words are defined as one of the following four types: lexicon words, mor- phologically derived words, factoids, and named entities. Our system provides a unified approach to the four fundamental features of word-level Chinese language processing: (1) word segmentation, (2) morphological analy- sis, (3) factoid detection, and (4) named entity recognition. The performance of the system is evaluated on a manually annotated test set, and is also compared with several state-of- the-art systems, taking into account the fact that the definition of Chinese words often varies from system to system. 1 Introduction Chinese word segmentation is the initial step of many Chinese language processing tasks, and has attracted a lot of attention in the research commu- nity. It is a challenging problem due to the fact that there is no standard definition of Chinese words. In this paper, we define Chinese words as one of the following four types: entries in a lexicon, mor- phologically derived words, factoids, and named entities. We then present a Chinese word segmen- tation system which provides a solution to the four fundamental problems of word-level Chinese lan- guage processing: word segmentation, morpho- logical analysis, factoid detection, and named entity recognition (NER). There are no word