基于局部敏感哈希的多维海量数据处理
|
摘要
针对多维海量的超精密加工机床状态监控数据难以被高效地存储与查询这一问题,文章提出了基于局部敏感哈希的多维海量数据处理方法。该方法利用P稳定的局部敏感哈希算法,一方面对数据进行散列化存储,使分散在各存储节点上的数据在存取时避免了读写热点;另一方面也实现了数据降维,通过其结果的碰撞操作,保证了各存储节点内数据具有一定的近邻性,这一性质以牺牲一定的查询准确率为代价极大地缩小了查询范围,从而间接地提高了查询效率。实验结果表明,该处理方法可以有效的提高多维海量数据的存储与查询效率。
In order to solve the problem that it is difficult to efficiently store and query the condition monitoring data of multidimensional and massive ultra-precision machining machine tools,a method of multi-dimensional massive data processing based on local sensitive Hash is proposed in this paper.In this method,P-stable local sensitive Hash algorithm is used,on the one hand,the data is hashed and stored,so that the data scattered on each storage node can avoid reading and writing hotspots,and on the other hand,the dimension reduction of the data is also realized.Through the collision operation of the results,the data in each storage node has a certain degree of adjacency,which greatly reduces the query range at the expense of certain query accuracy,and thus indirectly improving the query efficiency.The experimental results show that the method can effectively improve the efficiency of multidimensional massive data storage and query.
In order to solve the problem that it is difficult to efficiently store and query the condition monitoring data of multidimensional and massive ultra-precision machining machine tools,a method of multi-dimensional massive data processing based on local sensitive Hash is proposed in this paper.In this method,P-stable local sensitive Hash algorithm is used,on the one hand,the data is hashed and stored,so that the data scattered on each storage node can avoid reading and writing hotspots,and on the other hand,the dimension reduction of the data is also realized.Through the collision operation of the results,the data in each storage node has a certain degree of adjacency,which greatly reduces the query range at the expense of certain query accuracy,and thus indirectly improving the query efficiency.The experimental results show that the method can effectively improve the efficiency of multidimensional massive data storage and query.
引文
[1]何文强,王波,乔政,等.辊筒模具超精密加工机床液体静压导轨热变形分析[J].航空精密制造技术,2017,53(04):5-10.
[2]葛微,罗圣美,周文辉,等.HiBase:一种基于分层式索引的高效HBase查询技术与系统[J].计算机学报,2016,39(01):140-153.
[3]熊安萍,黄容,邹洋.一种基于混合索引的HDFS小文件存储策略[J].重庆邮电大学学报(自然科学版),2015,27(01):97-102.
[4]肖子达,朱立谷,冯东煜,等.分布式数据库聚合计算性能优化[J].计算机应用,2017,37(05):1251-1256.
[5]史世泽.局部敏感哈希算法的研究[D].西安电子科技大学,2013.
[2]葛微,罗圣美,周文辉,等.HiBase:一种基于分层式索引的高效HBase查询技术与系统[J].计算机学报,2016,39(01):140-153.
[3]熊安萍,黄容,邹洋.一种基于混合索引的HDFS小文件存储策略[J].重庆邮电大学学报(自然科学版),2015,27(01):97-102.
[4]肖子达,朱立谷,冯东煜,等.分布式数据库聚合计算性能优化[J].计算机应用,2017,37(05):1251-1256.
[5]史世泽.局部敏感哈希算法的研究[D].西安电子科技大学,2013.