基于Hadoop平台的数据分析方案的设计应用.pdfVIP

中国科技论文在线 基于Hadoop平台的数据分析方案的设计应 用 1,3 1,3 2* 姜文 ，辛阳 ，陈林顺 （1. 北京邮电大学网络与信息攻防技术教育部重点实验室，北京 100876； 2. 北京安码科技有限公司, 北京 100876； 3. 灾备技术国家工程实验室，北京100876） 摘要：面对互联网上的海量数据，单台主机已无法满足其存储和计算要求，分布式存储和分 布式计算的应用成为必然的趋势。其中 Hadoop 是应用较多的分布式存储和计算框架之一。 本文在该平台下，通过对国内某有哪些信誉好的足球投注网站引擎两个月内的上千万条用户有哪些信誉好的足球投注网站日志进行数据统计分 析，给出相应 Map/Reduce程序的设计思路和实例，并提出 Map/Reduce分布式程序的部分设 计和性能优化方法，实验结果表明，本文提出的这些方法能简化 Map/Reduce 程序设计、有 效提高程序性能。 关键词：分布式计算；Map/Reduce；文本处理；数据分析 中图分类号：TP319 Designed Application of the Scheme of Data-Analysis based on Hadoop Platform Jiang Wen1,3, Xin Yang1,3, Chen Linshun2 (1. Key Laboratory of network and information attack and defence technology of Ministry of Education, Beijing University of Posts and Telecommunications, Beijing 100876; 2. Beijing Safe-Code Technology Co.,Ltd, Beijing 100876; 3. National Engineering Laboratory for Disaster Backup and Recovery, Beijing 100876) Abstract: To face of massive data on the Internet, a single host has been unable to meet the storage and computing requirements, distributed storage and distributed computing is an inevitable trend, Hadoop is the most popular of them. In this paper, we do data-statistic-analysis in Sogou search engine, which contains 10 million search logs with two moths, and shows the method and instance of programming design patterns with Map/Reduce framework, at last we propose some method of program performance Optimization in Map/Reduce.Experiment results show that these methods can simplify the design of Map/Reduce programming, and improve performance effectively. Keywords:distributed computing; Map/Reduce; text-processing; data-analysis 0 引言 互联网高速发展，信息量不断膨胀，各种大型有哪些信誉好的足球投注网站引擎的访问量迅速增加，与此同时这 些大型系统都记录下了海量的