概率神经网络的结构优化研究及其应用
|
摘要
D.F.Specht提出的概率神经网络(Probabilistic Neural Network, PNN)是基于贝叶斯决策理论与Parzen窗概率密度估计方法而建立的一种分类网络。PNN的训练过程简单,算法容易设计,在模式识别及模式分类领域有着广泛的应用。然而,PNN拓扑结构的复杂性和训练样本的数目成直接比例关系,每个训练样本对应一个隐层神经元,当训练样本数量巨大时,将导致规模庞大的网络结构,从而阻碍了PNN网络的推广和应用。本文的研究工作主要集中于PNN网络结构优化方面。
论文首先对PNN的理论基础和网络结构模型进行研究,并分析了PNN分类网络的优缺点。PNN在运算过程中通过Parzen窗估计法得到类条件概率密度,根据贝叶斯决策提供对样本的分类。这种基于统计原理的神经网络模型无需训练样本的连接权值,由给定样本直接构成隐层,是完全前向的计算过程,训练简洁;同时样本量的增多却对PNN的设计实现带来了难度。
然后归纳总结了目前PNN拓扑结构优化的研究技术,包括样本向量的降维技术和PNN网络隐层神经元的选择算法。在此基础上提出了一种基于有监督信号的竞争学习算法,利用PNN网络的分类结果来调整隐中心矢量。具体工作包括混叠类别的PNN决策边界分析,算法的具体实施步骤,收敛条件的确定和平滑因子的选取。而且设计了一个对服从标准正态分布的二维空间样本进行分类的实验,实验结果表明此有监督竞争学习算法对服从标准正态分布的样本向量有较好的分类效果。
最后设计和实现了一个基于PNN网络模型的垃圾邮件过滤系统,利用精确率,误报率和漏报率对实验结果进行评价,并对结果进行了分析和比较。实验证明了提出的有监督竞争学习算法在PNN分类中具有较高的分类性能并且有效的降低了PNN拓扑结构的复杂性。
Probabilistic neural network(PNN) is a classification network, which is based on Bayesian decision theory and Parzen window method of probability function estimation. As the learning rule is simple and the algorithm is easy to design, PNN has a wide range of applications in the field of pattern recognition and pattern classification. However, the complexity of PNN’s topology structure is directly proportional to the number of training samples. Each training sample will determine one hidden neuron. The huge number of samples will lead to a large-scale structure of the network, thus limiting the promotion and application of PNN network. In this paper, the research focused on PNN structure optimization.
First of all, PNN’s theoretical basis and structure of the network model were studied in this thesis. In addition, we analyzed the advantages and disadvantages of the PNN classification network. Conditional probability density is estimated by Parzen window method in PNN’s computing process, and then samples are classified according to the Bayesian decision. The training process is simple for that the hidden layer of PNN is directly constructed by given samples. However, the large number of samples will make PNN difficult to design.
Secondly, we summed up the current optimization technologies of PNN topology structure, including dimension-reducing technique of sample vector and selection algorithm of PNN hidden neurons. On this basis, a new supervised competitive learning algorithm is developed: classification result of the PNN network is employed to adjust the location of hidden central vector. The detailed work contains an analysis of PNN’s decision boundary for the case of overlapping categories, concrete design of algorithm, determination of convergence condition and a method to select smoothing parameters. Further more, an experiment was carried out to classify two overlapping classes with elements normally distributed in the 2-dimentional plane. Simulations results show that PNN classifier trained by the new supervised competitive learning algorithm plays a better performance for normally distributed samples.
In the end, PNN classification network is used to design and implement a spam filtering system. The results are evaluated by three methods which are precise rate, false positive detection rate and false negative detection rate, the analysis and comparison of the results are also done. The experimental results indicate that the proposed algorithm for the PNN classification network has a better classification performance and reduces the complexity of the network structure effectively.
论文首先对PNN的理论基础和网络结构模型进行研究,并分析了PNN分类网络的优缺点。PNN在运算过程中通过Parzen窗估计法得到类条件概率密度,根据贝叶斯决策提供对样本的分类。这种基于统计原理的神经网络模型无需训练样本的连接权值,由给定样本直接构成隐层,是完全前向的计算过程,训练简洁;同时样本量的增多却对PNN的设计实现带来了难度。
然后归纳总结了目前PNN拓扑结构优化的研究技术,包括样本向量的降维技术和PNN网络隐层神经元的选择算法。在此基础上提出了一种基于有监督信号的竞争学习算法,利用PNN网络的分类结果来调整隐中心矢量。具体工作包括混叠类别的PNN决策边界分析,算法的具体实施步骤,收敛条件的确定和平滑因子的选取。而且设计了一个对服从标准正态分布的二维空间样本进行分类的实验,实验结果表明此有监督竞争学习算法对服从标准正态分布的样本向量有较好的分类效果。
最后设计和实现了一个基于PNN网络模型的垃圾邮件过滤系统,利用精确率,误报率和漏报率对实验结果进行评价,并对结果进行了分析和比较。实验证明了提出的有监督竞争学习算法在PNN分类中具有较高的分类性能并且有效的降低了PNN拓扑结构的复杂性。
Probabilistic neural network(PNN) is a classification network, which is based on Bayesian decision theory and Parzen window method of probability function estimation. As the learning rule is simple and the algorithm is easy to design, PNN has a wide range of applications in the field of pattern recognition and pattern classification. However, the complexity of PNN’s topology structure is directly proportional to the number of training samples. Each training sample will determine one hidden neuron. The huge number of samples will lead to a large-scale structure of the network, thus limiting the promotion and application of PNN network. In this paper, the research focused on PNN structure optimization.
First of all, PNN’s theoretical basis and structure of the network model were studied in this thesis. In addition, we analyzed the advantages and disadvantages of the PNN classification network. Conditional probability density is estimated by Parzen window method in PNN’s computing process, and then samples are classified according to the Bayesian decision. The training process is simple for that the hidden layer of PNN is directly constructed by given samples. However, the large number of samples will make PNN difficult to design.
Secondly, we summed up the current optimization technologies of PNN topology structure, including dimension-reducing technique of sample vector and selection algorithm of PNN hidden neurons. On this basis, a new supervised competitive learning algorithm is developed: classification result of the PNN network is employed to adjust the location of hidden central vector. The detailed work contains an analysis of PNN’s decision boundary for the case of overlapping categories, concrete design of algorithm, determination of convergence condition and a method to select smoothing parameters. Further more, an experiment was carried out to classify two overlapping classes with elements normally distributed in the 2-dimentional plane. Simulations results show that PNN classifier trained by the new supervised competitive learning algorithm plays a better performance for normally distributed samples.
In the end, PNN classification network is used to design and implement a spam filtering system. The results are evaluated by three methods which are precise rate, false positive detection rate and false negative detection rate, the analysis and comparison of the results are also done. The experimental results indicate that the proposed algorithm for the PNN classification network has a better classification performance and reduces the complexity of the network structure effectively.
引文
1 Simon Haykin.神经网络原理.叶世伟,史忠植译.第二版.机械工业出版社, 2006: 1~22
2 Specht.D.F. Probability Neural Networks. Neural Networks. 1990, 3: 109~118
3曾明发.反垃圾邮件调查报告发布及形势分析. 2006中国反垃圾邮件年会. 2006: 15~21
4 Pietro Burrascano. Learning Vector Quantization for the Probabilistic Neural Network. IEEE Trans on Neural Network. 1991, 2 (4): 458~461
5 Mao K Z, Tan K C, Ser W. Probabilistic Neural-network Structure Determination for Pattern Classification [J]. IEEE Transactions on Neural Networks. 2000, 11(4): 1009~1016
6 Streit, R.L., Luginbuhl, T.E. Maximum Likelihood Training of Probabilistic Neural Networks. IEEE Transactions on Neural Networks. 1994, 5(5): 764~783
7 Way-Soong Lim, M.V.C.Rao. A New Method of Reducing Network Complexity in Probabilistic Neural Network for Target Identification. IEICE Electronics Express. 2004, 1(17): 534~539
8蔡曲林.基于概率神经网络的模式识别.国防科学技术大学硕士学位论文. 2006: 19~27
9赵温波,杨鹭怡,王立明.径向基概率神经网络的混合结构优化算法.系统仿真学报. 2004, 16(10): 2175~2181
10 Ji-Xiang Du, De-Shuang Huang, Guo-Jun Zhang, Zeng-Fu Wang. A Novel Full Structure Optimization Algorithm for Radial Basis Probabilistic Neural Networks. Neurocomputing. 2006, 70 (1): 592~596
11邢杰,萧德云.基于PCA的概率神经网络结构优化.清华大学学报(自然科学版). 2008, 48(1): 141~144
12 Ning Zong, Xia Hong. A Multi-Level Probabilistic Neural Network. Electronic Edition. 2007: 516~525
13 Michael R. Berthold, Jay Diamond. Constructive Training of Probabilistic Neural Networks. Neurocomputing. 1998, 19: 167~183
14 Leszek Rutkowski, Senior Member. Adaptive Probabilistic Neural Networks for PatternClassification in Time-Varying Environment. IEEE Transactions on Neural Networks. 2004, 15(4): 811~827
15 G..Pajares, J.M.delacruz. A Probabilistic Neural Network for Attribute Selection in Stereovision Matching. Neural Computing. 2002, 11: 83~89
16 Wang Cheng-Ru, Wang Jin-Jia, Lian Qiu-Sheng. A Novel PNN Classification for Speaker Identification.自动化学报. 2004, 30(3): 371~379
17 Klaus L.E.Kaiser, Stefan Pan.niculescu. Using Probabilistic Neural Networks to Model the Toxicity of Chemicals to the Fathead Minnow(Pimephales Promelas):a Study Based on 865 Compounds. Chemosphere. 1999, 38(14): 3237~3245.
18 Specht, D.F. Probabilistic Neural Networks and Polynomial Adaline as Complementary Techniques for Classification. IEEE Transactions on Neural Networks. 1990, 1(1): 111~121
19 MacKay.D.J.C. Bayesian Methods for Neural Networks: Theory and Applications. Neural Computation. 1995: 448~472
20 Berger J. Robust Bayesian Analysis:Sensitivity to the Prior. Journal of Statistical Planning and Inference. 1990, 25: 303~328
21 Parzen E. On Estimation of Probability Density Function and Mode. Annals of Mathematical Statistics. 1962, 33: 1065~1076
22 Pobert R.W. Duin. On the Choice of Smoothing Parameters for Parzen Estimators of Probability Density Functions. IEEE Trans on Computers. 1976, 11: 1175~1179
23赵佳白.概率神经网络的改进研究及其在股票预测上的应用.暨南大学硕士学位论文. 2007: 8~21
24 Richard O.Duda, Peter E.Hart, David G.Stork.模式分类.李宏东,姚天翔译.第二版.机械工业出版社, 2006: 141~143
25 Specht D. F. Probabilistic Neural Network and General Regression Neural Network. Fuzzy Logic and Neural Network Handbook, 1996: 301~344
26高秀梅,杨静宇,袁小华,杨健.广义主分量分析及人脸识别.计算机工程与应用. 2003, 39(11): 31~33
27熊汉春,贺前华,李海洲.一种PNN网络的EM训练算法.华南理工大学学报(自然科学版). 1998, 26(7): 25~32
28黄德双.基于PCA的概率神经网络模式分类方法.北京理工大学学报. 1996, 16(1): 69~74
29康涛.一种基于PCA和RS的文本特征抽取方法.现代电子技术. 2007, 10: 88~90
30 K. Krishna, M. Narasimha Murty. Genetic K-means Algorithm. IEEE Transactions on Systems, Man, and Cybernetics. 1999, 29(3): 433~439
31郑柏杰.基于划分的聚类算法研究.重庆大学硕士学位论文. 2005:27~36
32 Sheng Ma, Chuanyi Ji, James Farmer. An Efficient EM-based Training Algorithm for Fedforward Neural Networks. Neural Networks. 1997, 10(2): 243~256
33 Tom M. Mitchell.机器学习.曾华军,张银奎译.机械工业出版社, 2006: 136~140
34 Linde Y, Buzo A, Gray R M. An Algorithm for Vector Quantizer Design[J]. IEEE Trans Commun. 1980, 28(1): 84~95.
35 Farshid Delgosha1, Mohammad B.Menhaj. Fuzzy Probabilistic Neural Networks: A Practical Approach to the Implementation of Baysian Classifier. Electronic Edition. Fuzzy Days, Dortmund, 2001: 76~85
36 Rumelhart D E, Zipser D. Feature Discovery By Competitive Learning. Cognitive Science. 1985, 9(1): 75~112
37 Grossberg S. Competitive Learning: from Iterative Activation to Adaptive Resonance. Cognitive Science. 1987, 11(1): 23~63
38李军辉.中文邮件语料库建设.重庆大学硕士学位论文. 2006: 26~40
39林珊,宁国宁,赵之霖.中文分词在邮件过滤系统中的应用.华南理工大学学报(自然科学版). 2004, 32(增刊): 112~116
40 G. Salton, A. Wong, C. S. Yang. A Vector Space Model for Automatic Indexing. Communications of the ACM. 1975, 18(11): 613~620
41张世荣.支持向量机文本分类算法研究.大连理工大学硕士学位论文. 2007: 7~17
42 Chen Yu, Zhuang Tiange. Mutual Information Based on Registration for 3-D Non-rigid Medical Image. Journal of Shanghai Jiaotong University. 1999, 33(9): 1125~1127
43潘文峰.基于内容的垃圾邮件过滤研究.中国科学院研究生院.硕士学位论文. 2004: 21~38
44胡佳妮,徐蔚然.中文文本分类中的特征选择算法研究.光通信研究. 2005, (3): 44~46
45 N.C.Das, S.K.Mazumde. Constrained Nonlinear Programming: A Minimum Cross-entropy Algorithm. Eng. Optimization. ,1999, 31(4): 479~487
46张玉芳,彭时名,吕佳.基于文本分类TFIDF方法的改进与应用.计算机工程. 2006, 32(19): 76~78
47 Yang Yiming. Comparative Study on Feature Selection in Text Categor-ization. Proceedings of the 14th International Conference on Machine Learning, Na-shville: Morgan Kaufmann, 1997: 412~420
2 Specht.D.F. Probability Neural Networks. Neural Networks. 1990, 3: 109~118
3曾明发.反垃圾邮件调查报告发布及形势分析. 2006中国反垃圾邮件年会. 2006: 15~21
4 Pietro Burrascano. Learning Vector Quantization for the Probabilistic Neural Network. IEEE Trans on Neural Network. 1991, 2 (4): 458~461
5 Mao K Z, Tan K C, Ser W. Probabilistic Neural-network Structure Determination for Pattern Classification [J]. IEEE Transactions on Neural Networks. 2000, 11(4): 1009~1016
6 Streit, R.L., Luginbuhl, T.E. Maximum Likelihood Training of Probabilistic Neural Networks. IEEE Transactions on Neural Networks. 1994, 5(5): 764~783
7 Way-Soong Lim, M.V.C.Rao. A New Method of Reducing Network Complexity in Probabilistic Neural Network for Target Identification. IEICE Electronics Express. 2004, 1(17): 534~539
8蔡曲林.基于概率神经网络的模式识别.国防科学技术大学硕士学位论文. 2006: 19~27
9赵温波,杨鹭怡,王立明.径向基概率神经网络的混合结构优化算法.系统仿真学报. 2004, 16(10): 2175~2181
10 Ji-Xiang Du, De-Shuang Huang, Guo-Jun Zhang, Zeng-Fu Wang. A Novel Full Structure Optimization Algorithm for Radial Basis Probabilistic Neural Networks. Neurocomputing. 2006, 70 (1): 592~596
11邢杰,萧德云.基于PCA的概率神经网络结构优化.清华大学学报(自然科学版). 2008, 48(1): 141~144
12 Ning Zong, Xia Hong. A Multi-Level Probabilistic Neural Network. Electronic Edition. 2007: 516~525
13 Michael R. Berthold, Jay Diamond. Constructive Training of Probabilistic Neural Networks. Neurocomputing. 1998, 19: 167~183
14 Leszek Rutkowski, Senior Member. Adaptive Probabilistic Neural Networks for PatternClassification in Time-Varying Environment. IEEE Transactions on Neural Networks. 2004, 15(4): 811~827
15 G..Pajares, J.M.delacruz. A Probabilistic Neural Network for Attribute Selection in Stereovision Matching. Neural Computing. 2002, 11: 83~89
16 Wang Cheng-Ru, Wang Jin-Jia, Lian Qiu-Sheng. A Novel PNN Classification for Speaker Identification.自动化学报. 2004, 30(3): 371~379
17 Klaus L.E.Kaiser, Stefan Pan.niculescu. Using Probabilistic Neural Networks to Model the Toxicity of Chemicals to the Fathead Minnow(Pimephales Promelas):a Study Based on 865 Compounds. Chemosphere. 1999, 38(14): 3237~3245.
18 Specht, D.F. Probabilistic Neural Networks and Polynomial Adaline as Complementary Techniques for Classification. IEEE Transactions on Neural Networks. 1990, 1(1): 111~121
19 MacKay.D.J.C. Bayesian Methods for Neural Networks: Theory and Applications. Neural Computation. 1995: 448~472
20 Berger J. Robust Bayesian Analysis:Sensitivity to the Prior. Journal of Statistical Planning and Inference. 1990, 25: 303~328
21 Parzen E. On Estimation of Probability Density Function and Mode. Annals of Mathematical Statistics. 1962, 33: 1065~1076
22 Pobert R.W. Duin. On the Choice of Smoothing Parameters for Parzen Estimators of Probability Density Functions. IEEE Trans on Computers. 1976, 11: 1175~1179
23赵佳白.概率神经网络的改进研究及其在股票预测上的应用.暨南大学硕士学位论文. 2007: 8~21
24 Richard O.Duda, Peter E.Hart, David G.Stork.模式分类.李宏东,姚天翔译.第二版.机械工业出版社, 2006: 141~143
25 Specht D. F. Probabilistic Neural Network and General Regression Neural Network. Fuzzy Logic and Neural Network Handbook, 1996: 301~344
26高秀梅,杨静宇,袁小华,杨健.广义主分量分析及人脸识别.计算机工程与应用. 2003, 39(11): 31~33
27熊汉春,贺前华,李海洲.一种PNN网络的EM训练算法.华南理工大学学报(自然科学版). 1998, 26(7): 25~32
28黄德双.基于PCA的概率神经网络模式分类方法.北京理工大学学报. 1996, 16(1): 69~74
29康涛.一种基于PCA和RS的文本特征抽取方法.现代电子技术. 2007, 10: 88~90
30 K. Krishna, M. Narasimha Murty. Genetic K-means Algorithm. IEEE Transactions on Systems, Man, and Cybernetics. 1999, 29(3): 433~439
31郑柏杰.基于划分的聚类算法研究.重庆大学硕士学位论文. 2005:27~36
32 Sheng Ma, Chuanyi Ji, James Farmer. An Efficient EM-based Training Algorithm for Fedforward Neural Networks. Neural Networks. 1997, 10(2): 243~256
33 Tom M. Mitchell.机器学习.曾华军,张银奎译.机械工业出版社, 2006: 136~140
34 Linde Y, Buzo A, Gray R M. An Algorithm for Vector Quantizer Design[J]. IEEE Trans Commun. 1980, 28(1): 84~95.
35 Farshid Delgosha1, Mohammad B.Menhaj. Fuzzy Probabilistic Neural Networks: A Practical Approach to the Implementation of Baysian Classifier. Electronic Edition. Fuzzy Days, Dortmund, 2001: 76~85
36 Rumelhart D E, Zipser D. Feature Discovery By Competitive Learning. Cognitive Science. 1985, 9(1): 75~112
37 Grossberg S. Competitive Learning: from Iterative Activation to Adaptive Resonance. Cognitive Science. 1987, 11(1): 23~63
38李军辉.中文邮件语料库建设.重庆大学硕士学位论文. 2006: 26~40
39林珊,宁国宁,赵之霖.中文分词在邮件过滤系统中的应用.华南理工大学学报(自然科学版). 2004, 32(增刊): 112~116
40 G. Salton, A. Wong, C. S. Yang. A Vector Space Model for Automatic Indexing. Communications of the ACM. 1975, 18(11): 613~620
41张世荣.支持向量机文本分类算法研究.大连理工大学硕士学位论文. 2007: 7~17
42 Chen Yu, Zhuang Tiange. Mutual Information Based on Registration for 3-D Non-rigid Medical Image. Journal of Shanghai Jiaotong University. 1999, 33(9): 1125~1127
43潘文峰.基于内容的垃圾邮件过滤研究.中国科学院研究生院.硕士学位论文. 2004: 21~38
44胡佳妮,徐蔚然.中文文本分类中的特征选择算法研究.光通信研究. 2005, (3): 44~46
45 N.C.Das, S.K.Mazumde. Constrained Nonlinear Programming: A Minimum Cross-entropy Algorithm. Eng. Optimization. ,1999, 31(4): 479~487
46张玉芳,彭时名,吕佳.基于文本分类TFIDF方法的改进与应用.计算机工程. 2006, 32(19): 76~78
47 Yang Yiming. Comparative Study on Feature Selection in Text Categor-ization. Proceedings of the 14th International Conference on Machine Learning, Na-shville: Morgan Kaufmann, 1997: 412~420