数据密集型计算中负载均衡的数据布局方法-北京邮电大学学报.pdfVIP

2013年8月 北 京 邮 电 大 学 学 报 Aug.2013 第36卷 第4期 Journal of Beijing University of Posts and Telecommunications Vol.36 No.4 摇 摇 文章编号:1007鄄5321(2013)04鄄0076鄄05 DOI:10.13190/ jbupt.201304.76.songj 数据密集型计算中负载均衡的数据布局方法 宋摇 杰,摇 李甜甜,摇 闫振兴,摇 朱志良 (东北大学 软件学院,沈阳110819) 摘要:广泛用于数据密集型计算的MapReduce模型将计算部署到数据端并行执行,数据布局将不再只影响存储本 身,还影响计算效率;节点上存储数据的特征决定该节点上任务的执行效率,负载均衡从传统的服务器管理或任务 调度研究转变成为以提高并行性为目的的数据布局研究,为此,分析了数据密集型计算和MapReduce环境中数据 布局的特点,提出了负载均衡的数据布局目标,并提出在特定环境下实现负载均衡的数据布局方法,最后通过实验 证明了数据布局目标和数据布局方法的有效性. 理论和实验结果证明,新提出的布局方法能有效地提高MapRe鄄 duce应用的并行性,优化其执行效率. 关摇 键摇 词:数据密集型计算;数据布局;负载均衡;MapRedcue;云计算 中图分类号:TP393摇 摇 摇 摇 文献标志码:A Load鄄Balanced Data Layout Approach in Data鄄Intensive Computing SONGJie,摇 LI Tian鄄tian,摇 YAN Zhen鄄xing,摇 ZHU Zhi鄄liang (Software College,Northeastern University,Shenyang 110819,China) Abstract:Widely usedin data鄄intensivecomputing,the MapReducemodel deployscomputingtotheda鄄 ta side so as to execute in parallel. On this occasion,data layout will not only affect the storage itself, but also affect the computing efficiency. Computing efficiency of node is determined by features of data stored on this node. Therefore,the study on load balancing is accordingly shifted from traditional server management or task scheduling to study of datalayout asapurposeto improveparallelism. The datalay鄄 out characteristics in data鄄intensive computing and MapReduce environment isanalyzed,aload鄄balanced goal of data layout is proposed,and a load鄄balanced data layout approach in a specific environment is presented aswell. The proposed datalayout goal and approach areproved effectivethrough experiments. It is shown that the proposed data layout approach can effectively improve the paralleli