Learning Probabilities From Random Observables in High Dimensions: The Maximum Entropy Distribution and Others
详细信息 查看全文
- 作者：Tomoyuki Obuchi ; Simona Cocco ; Rémi Monasson
- 关键词：Probabilistic inference ; Maximum entropy principle ; Replica method
- 刊名：Journal of Statistical Physics
- 出版年：2015
- 出版时间：November 2015
- 年：2015
- 卷：161
- 期：3
- 页码：598-632
- 全文大小：1,205 KB
- 参考文献：1.Jaynes, E.T.: Information theory and statistical mechanics. Phys. Rev. 106, 620-30 (1957)MATH MathSciNet CrossRef ADS
2.Jaynes, E.T.: Information theory and statistical mechanics II. Phys. Rev. 108, 171-90 (1957)MathSciNet CrossRef ADS
3.Balian, R.: Statistical mechanics and the maximum entropy method. In: Grassberger, P., Nadal, J.P. (eds.) From Statistical Physics to Statistical Inference and Back. NATO ASI series 428, pp. 11-3. Kluwer Academics Publisher, New York (1994)CrossRef
4.Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley series in telecommunications and signal processing. Wiley, New York (1991)MATH CrossRef
5.Jaynes, E.T.: On the rationale of maximum entropy methods. Proc. IEEE 70, 939-52 (1982)CrossRef ADS
6.Jaynes, E.T.: Monkeys, kangaroos and \(N\) . In: Justice, J.H. (ed.) Maximum-Entropy and Bayesian Methods in Applied Statistics, pp. 26-8. Cambridge University Press, Cambridge (1986)CrossRef
7.Tikochinsky, Y., Tishby, N.Z., Levine, R.D.: Alternative approach to maximum-entropy inference. Phys. Rev. A 30, 2638-644 (1984)CrossRef ADS
8.Bialek, W.: Biophysics: Searching for Principles. Princeton University Press, Princeton (2012)
9.De Almeida, J.R.L., Thouless, D.J.: Stability of the Sherrington-Kirkpatrick solution of a spin glass model. J. Phys. A 11, 983-90 (1978)CrossRef ADS
10.Obuchi, T., Monasson, R.: J. Phys. Conf. Ser (2015) (Submitted to) - 作者单位：Tomoyuki Obuchi (1)
Simona Cocco (2)
Rémi Monasson (2)
1. Department of Computational Intelligence and Systems Science, Tokyo Institute of Technology, Yokohama, Japan
2. Laboratoire de Physique Statistique de l’Ecole Normale Supérieure, affilié au CNRS & à l’Université Pierre et Marie Curie, Paris, France - 刊物类别：Physics and Astronomy
- 刊物主题：Physics
Statistical Physics
Mathematical and Computational Physics
Physical Chemistry
Quantum Physics - 出版者：Springer Netherlands
- ISSN：1572-9613
文摘
We consider the problem of learning a target probability distribution over a set of N binary variables from the knowledge of the expectation values (with this target distribution) of M observables, drawn uniformly at random. The space of all probability distributions compatible with these M expectation values within some fixed accuracy, called version space, is studied. We introduce a biased measure over the version space, which gives a boost increasing exponentially with the entropy of the distributions and with an arbitrary inverse ‘temperature-\(\Gamma \). The choice of \(\Gamma \) allows us to interpolate smoothly between the unbiased measure over all distributions in the version space (\(\Gamma =0\)) and the pointwise measure concentrated at the maximum entropy distribution (\(\Gamma \rightarrow \infty \)). Using the replica method we compute the volume of the version space and other quantities of interest, such as the distance R between the target distribution and the center-of-mass distribution over the version space, as functions of \(\alpha =(\log M)/N\) and \(\Gamma \) for large N. Phase transitions at critical values of \(\alpha \) are found, corresponding to qualitative improvements in the learning of the target distribution and to the decrease of the distance R. However, for fixed \(\alpha \), the distance R does not vary with \(\Gamma \), which means that the maximum entropy distribution is not closer to the target distribution than any other distribution compatible with the observable values. Our results are confirmed by Monte Carlo sampling of the version space for small system sizes (\(N\le 10\)). Keywords Probabilistic inference Maximum entropy principle Replica method