针对数据存储安全性差、数据传输服务器负载压力大等问题,本文设计了海量工业大数据并行传输与扩容存储系统。并行传输子系统通过用户界面模块、无线通信模块以及USB模块等相互协作,采用FPGA并行传输技术,基于Cyclone V系列核心处理器,将字符编码后以P2P形式进行并行高速传输;扩容存储子系统通过整合dedup与map/reduce技术删除接口采集的重复数据,利用大数据安全容错算法控制数据的收敛程度与弹性,实现大数据扩容存储。实验结果证明,系统的大数据存储完整度较高;能够实现大数据稳定并行传输;数据存储安全性高,且在大数据并行传输时,服务器实时负载压力始终未超过20%。

Aiming at the problems of poor data storage security and high load pressure on data transmission server,a massive industrial big data parallel transmission and capacity expansion storage system is designed.The parallel transmission subsystem cooperates with each other through user interface module,wireless communication module and USB module,adopts FPGA parallel transmission technology,and carries out parallel high-speed transmission in the form of P2P after character coding based on Cyclone V series core processor.The capacity expansion storage subsystem removes the duplicate data collected by the interface by integrating dedup and map/reduce technology,and uses the big data security fault-tolerant algorithm to control the convergence and elasticity of data to realize the capacity expansion storage of big data.The experiment results show that the big data storage integrity of the system is high.It can realize stable parallel transmission of big data.The security of data storage is high, and the real-time load pressure of the server does not exceed 20% when large data is transmitted in parallel.