一种利用并发提高数据处理吞吐率的模型.PDFVIP

计 算 机 系 统 应 用 2010 第 19卷 第 11 期 一种利用并发提高数据处理吞吐率的模型① 周钦强 (广东省大气探测技术中心 广东 广州 510080) 要: 针对分布式组网终端集中式数据采集与处理系统实时性高、数据吞吐量大、周期性强等特点，引入了 一种利用并发机制的大吞吐量实时数据处理模型，在讨论模型基本结构和原理的基础上，深入研究了 决定模型数据吞吐效率和时效性能的并发缓存机制和算法设计。利用该模型实现的数据采集处理系统 在基于GPRS 组网的1500 个自动气象站数据采集与处理业务应用中，性能稳定可靠，能够满足对周 期性大批量实时 “浪涌”数据进行实时快速处理的要求，大大提高了数据处理的吞吐效率。 关键词: 并发; 数据处理; 实时; 吞吐量; 模型 Real-Time and Large Throughput Data Processing System Using Concurrency ZHOU Qin-Qiang (Air Observing Center of Guangdong Meteorology, Guangzhou GuangDong 510080, China) Abstract: Data acquiring and processing from distributed terminal unit by network are usually real-time, large throughput and periodic, so data acquisition, processing and result should be done within data output period in high-efficiency. A real-time and large throughput data processing model using concurrency is built for this given application. The working principle and basic structure for the model are discussed in detail, also concurrency for the model and corresponding key buffer which concurrent threads put data into and gets data out are analyzed in particular. As is used in observation application networked by 1500 automatic weather stations, which has indicated that the model is capable in processing periodic surge observation data and has higher data throughput, as well as implements high speed and accurate results, high stability and reliability. Keywords: concurrency; data processing (DP); real-time; data throughput; model analysis 分布式数据采集终端具有分布式、实时、同步、 引入并发机制的实时数据处理模型，着力分析利 高密度、周期性输出等特点，数据采集往往通过无线 用并发机制提高数据采集处理吞吐效率的原理，重点 或有线方式集中采集到数据处理中心。基于 GPRS 通 讨论了决定并发效能的关键点--并发缓存与算法设 信技术的新一代自动气象站组