藏文网页定题采集方法分析-计算机应用技术专业论文.docxVIP

I I 摘 要 与汉文相比，藏文信息处理技术发展较慢，加之缺乏支持藏文有哪些信誉好的足球投注网站引擎，互 联网上的藏文信息常常处于“孤立状态”，给用户的查找和获取带来较大的困难。 因此，探讨一种通过网络采集藏文信息的方法，对于藏文研究者显得尤为重要。 在分析了网页采集流程、网络爬虫工作基本原理和主题网页采集的相关知识 的基础上，对藏文网页的采集方法进行了深入研究： 1．对比分析藏文网页的字体、藏文音节点、藏文高频词等区别于其他网页 的特征参数，设计出适合于判断藏文网页的相关算法。 2．探讨了藏文主题爬虫的关键技术，如藏文分词、主题判断方法以及爬虫 的爬行策略等内容，提出基于“导向词”的藏文主题判断方法。 3．研究 Heritrix 软件，并通过对其关键模块 Extractor 和 Frontierscheduler 的改进和扩展，实现“导向词”算法的藏文主题信息网站的抓取；另外，运用哈 希算法，扩展 Queue-assignment-policy 模块，大大提升了爬虫的采集效率。 4．利用 HTMLParse 软件对采集的新闻信息进行提取，并将新闻的标题、发 布时间、来源、正文信息存入数据库。 5．对采集的藏文网页文本进行编码“归一化”处理，转化成国际标准的 Unicode 编码。 利用上述研究结果，以网页的查准率和查全率为参考指标，对“导向词”主 题判断算法的几个阙值进行了测试，根据测试的结果对中国西藏网进行了网页抓 取，抓取的准确率在 62%左右。测试数据表明，研究结果对于藏文定题信息采集 行之有效，具有较高的应用和理论参考价值。 关键词：Web 检索，藏文网页采集，网络爬虫，定题采集，Heritrix II II Abstract Compared with the Chinese, Tibetan information processing technology development is slower. Because of a lack of Tibetan search engine, Tibetan information is still in isolation, and it brings great difficulty for users to search and get information from internet. So, It is thus very important to find a way to search and collect Tibetan information from internet for Tibetan studies. The present paper first introduced the procedure of webcollecting and some basic principles of web crawling. It then, with regard to Tibetan, proposed approaches of recognizing and collecting WebPages. The research steps are as follows: The paper analyzed fonts, syllables, high frequency words and some other specific featured parameters of Tibetan, and designed algorithm in accord with Tibetan WebPages. It then discussed key techniques in crawling, such as word-segregation, topic judgment, and crawling strategy, and further put forward a judging method making use of ―guide words‖. Based on the improvement and expansion of model ―Extractor‖ and ―Frontierscheduler‖ from Heritrix soft, the research realized the catching of WebPages in Tibetan. Besides, the use of Hash Algorithm and extension of Queue-assignment - policy also promoted the efficiency of crawling. The research e