基于节点属性与正文内容的海量web信息抽取方法information.pdfVIP

第 37 卷第 10 期 通 信 学 报 Vol.37 No.10 2016 年 10 月 Journal on Communications October 2016 doi:10.11959/j.issn.1000-436x.2016190 基于节点属性与正文内容的海量 Web 信息抽取方法 王海艳 1,2 ，曹攀 1 （1. 南京邮电大学计算机学院，江苏 南京 210023 ；2. 江苏省无线传感网高技术研究重点实验室，江苏 南京 210003 ） 摘 要：为解决大数据场景下从海量 Web 页面中抽取有价值的信息，提出了一种基于节点属性与正文内容的海量 Web 信息抽取方法。将 Web 页面转化为 DOM 树表示，并提出剪枝与融合算法，对 DOM 树进行简化；定义 DOM 树节点的密度和视觉属性，根据属性值对 Web 页面内容进行预处理；引入 MapReduce 计算框架，实现海量 Web 信息的并行化抽取。仿真实验结果表明，提出的海量 Web 信息抽取方法不仅具有更好的性能，还具备较好的系统 可扩展性。 关键词：Web 信息；抽取；MapReduce ；DOM 树 中图分类号：TP393.07 文献标识码：A Information extraction from massive Web pages based on node property and text content WANG Hai-yan1,2, CAO Pan1 (1. School of Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing 210023, China; 2. Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks, Nanjing 210003, China) Abstract: To address the problem of extracting valuable information from massive Web pages in big data environ- ments, a novel information extraction method based on node property and text content for massive Web pages was put forward. Web pages were converted into a document object model (DOM) tree, and a pruning and fusion algorithm was introduced to simplify the DOM tree. For each node in the DOM tree, both density property and vision property was defined and Web pages were pretreated based on these property values. A MapReduce framework was employed to realize parallel information extraction from massive Web pages. Simulation and experimental results demonstrate that the proposed extractio