| 摘 要: 由于数据流的不稳定性，将数据流查询安排在固定节点上就会造成分布式数据流处理技术很难对计算资 源实现较高的处理效率，基于此，提出大数据分析下分布式数据流处理技术研究。具体流程是数据收集、历史数据的存 储和查询、Storm实时处理、智能索引、数据模型的建立。根据实验结果可知，本文提出的大数据分析下分布式数据流 处理技术与传统技术相比，在数据流的处理效率上占有较大优势，一般维持在75%以上，能够大大节省处理时间。 |
| 关键词: 大数据；分布式；数据流处理技术；处理效率 |
|
中图分类号: TP333
文献标识码: A
|
|
| Research on Distributed Data Flow Processing Technology under Big Data Analysis |
|
LIU Qin
|
( School of Computer Science, Qinghai Nationalities University, Xining 810007, China)
|
| Abstract: Because of the instability of data flow,it is difficult for distributed data flow processing technology to achieve high processing efficiency for computing resources by arranging data flow query on fixed nodes.For this reason,this paper proposes the research of distributed data flow processing technology under big data analysis.The specific process is data collection,historical data storage and query,storm real-time processing,intelligent index,data model building.According to the experimental results,compared with the traditional technology,the distributed data flow processing technology proposed in this paper has a greater advantage in the efficiency of data flow processing,generally maintained at more than 75%,which can greatly save processing time. |
| Keywords: big data;distributed;data flow processing technology;processing efficiency |
