面向大数据环境的检测器快速筛选算法.docVIP

面向大数据环境的检测器快速筛选算法 蔡涛 倪晓蓉 王伟生 牛德姣 (江苏大学 计算机科学与通信工程学院 江苏镇江 212013caitao@ujs.edu.cn 摘要 筛选成熟检测器是决定人工免疫系统的性能和效率的关键因素，在大数据环境下由于初始检测器的数量极其庞大，造成现有检测器筛选算法时间开销过大的问题。本文针对海量的初始检测器，设计海量初始检测器的分布存储模式，为提高筛选初始检测器的效率提供基础；利用Map/Reduce模型，设计了新型的海量初始检测器快速筛选算法，给出了混合式初始检测器快速筛选架构、海量初始检测器分区检查策略和成熟检测器集优化策略，提高筛选初始检测器的效率，优化成熟检测器。最后在Hadoop集群中，实现了面向大数据环境检测器快速筛选算法原型系统，使用CERT synthethic sendmail data数据集进行了测试与分析，验证了算法能减少58.87%的时间开销，并在初始检测器数量不断增加时保持时间开销的稳定。 关键字 检测器生成算法，大数据系统，人工免疫算法，Map/Reduce 中图法分类号 TP391 文献标识码 A The Fast Detector Generation Algorithm for the Big Data System Cai Tao, Ni Xiaorong, Wang Weisheng, Niu Dejiao (School of Computer Science and Telecommunication Engineering, JiangSu University, Zhenjiang, Jiangsu 212013) Abstract The detector generation algorithm is important for the performance and efficiency of artificial immune system. But the time overhead of current detector generation algorithm is too big due to the large number of initial detector in the big data system. In this paper, the new distributed initial detector storage strategy is designed to provide the basis for improving the massive initial detector selection efficiency. Then, the new fast detector generation algorithm for the big data system is given by analyzing the different character of map and reduce stage in Map/reduce model, and it is used to improve the initial detector selection efficiency and optimize the mature detector set. It includes hybrid initial detector selection architecture, massive initial detection partition inspection algorithm and mature detector set optimization algorithm. At last, the fast detector generation algorithm prototype for big data system is realized in Hadoop system. CERT synthethic sendmail data set is used to evaluate and compare with the current detector generation algorithm. The result shows the 58.87% time overhead is reduced, and time overhead can been maintained stability with the number of initial detector increasing. Key Words Detector generation a