现代汉语文本的词语切分技术（Technique of word segmentation in modern Chinese text）.docVIP

现代汉语文本的词语切分技术（Technique of word segmentation in modern Chinese text） I. Introduction 1, the necessity of Chinese automatic word segmentation Chinese automatic word segmentation is the first step in automatic analysis of Chinese text. You can imagine the difficulties in the process of Chinese automatic segmentation: if all spaces of an English text symbols are removed, and then let the computer automatically restore the original text spaces, this is the word recognition process, the main problem of this process is to deal with ambiguity. The word reflects the Chinese and English is significantly different. English text is a small character set has been fully separated by words, while the Chinese text is a continuous string on large character set. The string is divided into words, is the automatic segmentation system need to do. Words are the smallest, independent, meaningful language components. All the language knowledge of computer from the machine dictionary (the information given words) and syntax (to describe the aggregation phenomenon of words in various combinations of semantic and lexical category), word and sentence context, pragmatic knowledge base. As long as the Chinese information processing system involves syntax and semantics (such as retrieval, translation, abstract and peer equivalence), it is necessary to use the word as the basic unit. For example, Pinyin, word conversion, simplified traditional Chinese translation, printed or handwritten Chinese characters recognition, automatic reading of Chinese text (speech synthesis), etc., all require the use of word information. After segmentation conversion or recognition at the word level, the certainty is improved greatly. In information retrieval, if not segmentation (by word retrieval), when retrieving German monetary unit Mark, Marx will be retrieved, and the retrieved Chinese when the Peoples Republic of China retrieved. If the segmentation, will greatly improve the accuracy of retrieval. In the highe