CloudResearch(云服务)_13.docVIP

Adapting MapReduce for Dynamic Environments Using a Peer-to-Peer Model Fabrizio Marozzo, Domenico Talia, Paolo Trunfio DEIS, University of Calabria, Via P. Bucci 41C, 87036 Rende, Italy fmarozzo@unical.it, ftalia,trunfiog@deis.unical.it Extended Abstract Introduction MapReduce is a programming model used for processing large data sets in a highly-parallel way [1]. Users specify the computation in terms of a “map” function that processes a key/value pair to generate a set of intermediate key/value pairs, and a “reduce” function that merges all intermediate values associated with the same intermediate key. MapReduce implementations (e.g., Google’s MapReduce [2] and Apache Hadoop [3]) are based on a master-slave model. A job is submitted by a user node to a master node that selects idle workers and assigns each one a map or a reduce task. When all map and reduce tasks have been completed, the master node returns the result to the user node. The failure of a worker is managed by re-executing its task on another worker, while master failures are not managed by current MapReduce implementations as designers consider failures unlikely in large clusters or in reliable Cloud environments. On the contrary, node failures (including master failures) can occur in large clus-ters and Clouds and are likely to happen in dynamic environments, like computa-tional Grids and volunteer computing systems, where nodes join and leave the net-work at an unpredictable rate. Therefore, providing e ective mechanisms to manage master failures is fundamental to exploit the MapReduce model in the implemen-tation of data-intensive applications in those dynamic environments where current MapReduce implementations could be unreliable. The goal of this work is investi-gating how to improve the master-slave architecture of current MapReduce imple-mentations to make it more suitable for Grid-like and P2P dynamic scenarios. The extended model we introduce here exploits a P2P model to dynamically assig