基于非时间属性关联的数据逼真生成算法
|
摘要
提出基于非时间属性关联的数据逼真生成算法.该算法可以解决数据生成器研发中非时间属性关联构建的困难问题,在大数据测评领域中对仿真数据生成有重要应用价值.首先,从数据集中提取关键的两个非时间属性,对它们分别做两重频数统计.然后,根据两次统计结果计算最大信息系数值来评估相关性,用拉伸指数分布进行拟合,构建出关联模型.最后,通过模型参数构建约束,在此约束的二维矩阵中生成数据.实验结果表明,该算法能够有效地模拟真实数据集的数据特征.
A table data simulation generating algorithm is proposed based on not-temporal attribute correlation. This algorithm can overcome the difficulty in building not-temporal attribute correlation in the development of big data simulation generator, and play an important role in the field of measurement of the big data simulation generated. Firstly,we extract the two key not-temporal attributes from the data set, and make the statistics of twofold frequency. Then, based on the statistical results, we calculate the maximal information coefficient(MIC) value to measure dependence for twovariable relationships. We use the stretched exponential(SE) distribution to fit the relationship, and build the correlation model. Finally, we generate data in a two-dimensional matrix with this model. The experimental results show that this algorithm can effectively describe the data characteristics of the real data set.
A table data simulation generating algorithm is proposed based on not-temporal attribute correlation. This algorithm can overcome the difficulty in building not-temporal attribute correlation in the development of big data simulation generator, and play an important role in the field of measurement of the big data simulation generated. Firstly,we extract the two key not-temporal attributes from the data set, and make the statistics of twofold frequency. Then, based on the statistical results, we calculate the maximal information coefficient(MIC) value to measure dependence for twovariable relationships. We use the stretched exponential(SE) distribution to fit the relationship, and build the correlation model. Finally, we generate data in a two-dimensional matrix with this model. The experimental results show that this algorithm can effectively describe the data characteristics of the real data set.
引文
1 Guo L,Tan EH,Chen SQ,et al.The stretched exponential distribution of internet media access patterns.Proceedings of the Twenty-Seventh ACM Symposium on Principles of Distributed Computing.Toronto,Canada.2008.283-294.
2韩筱璞,汪秉宏,周涛.人类行为动力学研究.复杂系统与复杂性科学,2010,7(2):132-144.
3 Guo L,Tan EH,Chen SQ,et al.Analyzing patterns of user content generation in online social networks.Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.Paris,France.2009.369-378.
4 Busari M,Williamson C.ProWGen:A synthetic workload generation tool for simulation evaluation of Web proxy caches.Computer Networks,2002,38(6):779-794.[doi:10.1016/S1389-1286(01)00285-7]
5 Ming ZJ,Luo CJ,Gao WL,et al.BDGS:A scalable big data generator suite in big data benchmarking.In:Rabl T,Raghunath N,Poess M,et al,eds.Advancing Big Data Benchmarks.Cham,Swizerland:Springer,2014.138-154.
6 Rabl T,Frank M,Sergieh HM,et al.A data generator for cloud-scale benchmarking.Proceedings of the Second TPC Technology Conference on Performance Evaluation,Measurement and Characterization of Complex Systems.Berlin,Heidelberg,Germany.2010.41-56.
7詹剑锋,高婉铃,王磊,等.BigDataBench:开源的大数据系统评测基准.计算机学报,2016,39(1):196-211.[doi:10.11897/SP.J.1016.2016.00196]
8 Gray J,Sundaresan P,Englert S,et al.Quickly generating billion-record synthetic databases.Proceedings of the 1994ACM SIGMOD International Conference on Management of Data.Minneapolis,MN,USA.1994.243-252.
9 Cooper BF,Silberstein A,Tam E,et al.Benchmarking cloud serving systems with YCSB.Proceedings of the 1st ACMSymposium on Cloud Computing.Indianapolis,IN,USA.2010.143-154.
10 Abramova V,Bernardino J,Furtado P.Evaluating Cassandra scalability with YCSB.International Conference on Database and Expert Systems Applications.Springer International Publishing 2014.199-207.
11 Yin JW,Lu XJ,Zhao XK,et al.BURSE:A bursty and selfsimilar workload generator for cloud computing.IEEE Transactions on Parallel and Distributed Systems,2015,26(3):668-680.[doi:10.1109/TPDS.2014.2315204]
12 Akrour N,Mallet C,Barthes L,et al.A rainfall simulator based on multifractal generator.EGU General Assembly Conference Abstracts.Vienna,Austria.2015.
13 Ansari N,Liu H,Shi YQ,et al.On modeling MPEG video traffics.IEEE Transactions on Broadcasting,2002,48(4):337-347.[doi:10.1109/TBC.2002.806794]
14 Jiang M,Nikolic M,Hardy S,et al.Impact of self-similarity on wireless data network performance.Proceedings of IEEE International Conference on Communications.Helsinki,Finland.2001.477-481.
15 Speed T.A correlation for the 21st century.Science,2011,334(6062):1502-1503.[doi:10.1126/science.1215894]
16 Fan JQ,Han F,Liu H.Challenges of big data analysis.National Science Review,2014,1(2):293-314.[doi:10.1093/nsr/nwt032]
17 Rabl T,Lang A,Hackl T,et al.Generating shifting workloads to benchmark adaptability in relational database systems.In:Nambiar R,Poess M,eds.Performance Evaluation and Benchmarking.Berlin Heidelberg,Germany:Springer,2009.116-131.
18钱宇华,成红红,梁新彦,等.大数据关联关系度量研究综述.数据采集与处理,2015,30(6):1147-1159.
19 Reshef DN,Reshef YA,Finucane HK,et al.Detecting novel associations in large data sets.Science,2011,334(6062):1518-1524.[doi:10.1126/science.1205438]
2韩筱璞,汪秉宏,周涛.人类行为动力学研究.复杂系统与复杂性科学,2010,7(2):132-144.
3 Guo L,Tan EH,Chen SQ,et al.Analyzing patterns of user content generation in online social networks.Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.Paris,France.2009.369-378.
4 Busari M,Williamson C.ProWGen:A synthetic workload generation tool for simulation evaluation of Web proxy caches.Computer Networks,2002,38(6):779-794.[doi:10.1016/S1389-1286(01)00285-7]
5 Ming ZJ,Luo CJ,Gao WL,et al.BDGS:A scalable big data generator suite in big data benchmarking.In:Rabl T,Raghunath N,Poess M,et al,eds.Advancing Big Data Benchmarks.Cham,Swizerland:Springer,2014.138-154.
6 Rabl T,Frank M,Sergieh HM,et al.A data generator for cloud-scale benchmarking.Proceedings of the Second TPC Technology Conference on Performance Evaluation,Measurement and Characterization of Complex Systems.Berlin,Heidelberg,Germany.2010.41-56.
7詹剑锋,高婉铃,王磊,等.BigDataBench:开源的大数据系统评测基准.计算机学报,2016,39(1):196-211.[doi:10.11897/SP.J.1016.2016.00196]
8 Gray J,Sundaresan P,Englert S,et al.Quickly generating billion-record synthetic databases.Proceedings of the 1994ACM SIGMOD International Conference on Management of Data.Minneapolis,MN,USA.1994.243-252.
9 Cooper BF,Silberstein A,Tam E,et al.Benchmarking cloud serving systems with YCSB.Proceedings of the 1st ACMSymposium on Cloud Computing.Indianapolis,IN,USA.2010.143-154.
10 Abramova V,Bernardino J,Furtado P.Evaluating Cassandra scalability with YCSB.International Conference on Database and Expert Systems Applications.Springer International Publishing 2014.199-207.
11 Yin JW,Lu XJ,Zhao XK,et al.BURSE:A bursty and selfsimilar workload generator for cloud computing.IEEE Transactions on Parallel and Distributed Systems,2015,26(3):668-680.[doi:10.1109/TPDS.2014.2315204]
12 Akrour N,Mallet C,Barthes L,et al.A rainfall simulator based on multifractal generator.EGU General Assembly Conference Abstracts.Vienna,Austria.2015.
13 Ansari N,Liu H,Shi YQ,et al.On modeling MPEG video traffics.IEEE Transactions on Broadcasting,2002,48(4):337-347.[doi:10.1109/TBC.2002.806794]
14 Jiang M,Nikolic M,Hardy S,et al.Impact of self-similarity on wireless data network performance.Proceedings of IEEE International Conference on Communications.Helsinki,Finland.2001.477-481.
15 Speed T.A correlation for the 21st century.Science,2011,334(6062):1502-1503.[doi:10.1126/science.1215894]
16 Fan JQ,Han F,Liu H.Challenges of big data analysis.National Science Review,2014,1(2):293-314.[doi:10.1093/nsr/nwt032]
17 Rabl T,Lang A,Hackl T,et al.Generating shifting workloads to benchmark adaptability in relational database systems.In:Nambiar R,Poess M,eds.Performance Evaluation and Benchmarking.Berlin Heidelberg,Germany:Springer,2009.116-131.
18钱宇华,成红红,梁新彦,等.大数据关联关系度量研究综述.数据采集与处理,2015,30(6):1147-1159.
19 Reshef DN,Reshef YA,Finucane HK,et al.Detecting novel associations in large data sets.Science,2011,334(6062):1518-1524.[doi:10.1126/science.1205438]