一个开源的藏文分词词性标注系统.pdfVIP

文章编号：1003-0077 （2011 ）00-0000-00 TIP-LAS:一个开源的藏文分词词性标注系统 李亚超，江静，加羊吉，于洪志 （甘肃省民族语言智能处理重点实验室（西北民族大学），甘肃 兰州 730030 ） 摘 要：TIP-LAS 是一个开源的藏文分词词性标注系统，提供藏文分词、词性标注功能。该系统基于条件 随机场模型实现基于音节标注的藏文分词系统，采用最大熵模型，并融合音节特征，实现藏文词性标注系 统。经过试验及对比分析，TIP-LAS 藏文分词系统词性标注系统取得了较好的实验效果，系统的源代码可 以从网上获取。希望该研究可以推动藏文分词、词性标注等基础工作的发展，提供一个可以比较、共享的 研究平台。 关键词：藏文；分词；词性标注；条件随机场；最大熵 中图分类号：TP391 文献标识码：A TIP-LAS: An Open Source Toolkit for Tibetan Word Segmentation and Part of Speech Tagging LI Yachao, JIANG Jing, JIA Yangji, YU Hongzhi (Key Laboratory of National Language Intelligent Processing (Northwest University for Nationalities), Lanzhou, Gansu 730030, China) Abstract: TIP-LAS is an open source toolkit for Tibetan segmentation and POS tagging, which has the function of Tibetan word segmentation and POS tagging. The toolkit implements the Tibetan segmentation system based on syllable tagging with the conditional random fields model, and integrates the maximum entropy model with syllables features to implement the Tibetan POS tagging system. By the experiment and comparative analysis, The System have achieved good experimental results and the source code can be obtained from the Internet, the experimental corpus have also been shared in the Internet. We are expected that the study can promote the development of Tibetan word segmentation, POS tagging and other infrastructure work, and can provide a research platform for comparison and sharing. Key words: Tibetan; word segmentation; part of speech tagging; conditional random fields; maximum entropy 1 引言 藏文是一种拼音文字，有30 个辅音字母和4 个元音字母，由这些字母组成音节，由音 节构成词，音节之间用音节点“་” （tsheg ）作为分隔符。藏语词语之间没有明显的分隔符来 进行区分，因此藏语信息处理首先要面对分词问题。虽然藏语是一种古老的语言，但是对于