On the Computability of Continuous Maximum Entropy Distributions with Applications
Abstract.
We study the following problem: Given a continuous domain \(\Omega\) along with its convex hull \(\mathcal{K}\) , a point \(A \in \mathcal{K}\) , and a measure \(\mu\) on \(\Omega\) , find the probability density over \(\Omega\) whose marginal is \(A\) and that minimizes the KL divergence to the uniform density with respect to \(\mu\) . Several distributions in mathematics, physics, statistics, and theoretical computer science arise by different settings of the parameters of this problem. We give a polynomial bound on the norm of the optimizer of the dual problem that holds in a very general setting and relies on a “balance” property of the measure \(\mu\) on \(\Omega\) , and exact algorithms for evaluating the dual and its gradient for several interesting settings of \(\Omega\) and \(\mu\) . Together, along with the ellipsoid method, these results imply polynomial-time algorithms to compute such KL divergence minimizing distributions in several cases. Applications of our results include (1) an optimization characterization of the Goemans–Williamson measure [M. X. Goemans and D. P. Williamson, J ACM, 42 (1995), pp. 1115–1145] that is used to round a positive semidefinite matrix to a vector; (2) the computability of the entropic barrier for convex bodies, given a strong integration oracle, studied by [S. Bubeck and R. Eldan, Proc. Mach. Learn. Res. (PMLR), 40 (2015), p. 279], and (3) a polynomial-time algorithm to compute the barycentric quantum entropy of a density matrix that was proposed as an alternative to von Neumann entropy [W. Band and J. L. Park, Found. Phys., 6 (1976), pp. 249–262; J. L. Park and W. Band, Found. Phys., 7 (1977), pp. 233–244; P. B. Slater, Phys. Lett. A, 159 (1991), pp. 411–414]; this corresponds to the case when \(\Omega\) is the set of rank-one projection matrices and \(\mu\) is derived from the Haar measure on the unit sphere.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Z. Allen-Zhu, Y. Li, R. Oliveira, and A. Wigderson, Much faster algorithms for matrix scaling, in FOCS’17: Proceedings of the 58th Annual IEEE Symposium on Foundations of Computer Science, IEEE, Piscataway, NJ, 2017, pp. 890–901.
2.
B. C. Arnold, Majorization and the Lorenz Order: A Brief Introduction, Lect. Notes Stat. 43, Springer, Berlin, 2012.
3.
A. Asadpour, M. X. Goemans, A. Madry, S. O. Gharan, and A. Saberi, An \(O (\log n/ \log \log n)\) -approximation algorithm for the asymmetric traveling salesman problem, Oper. Res., 65 (2017), pp. 1043–1061.
4.
W. Band and J. L. Park, New information-theoretic foundations for quantum statistics, Found. Phys., 6 (1976), pp. 249–262, https://doi.org/10.1007/BF00708800.
5.
A. Ben-Tal and A. Nemirovski, Optimization III: Convex Analysis, Nonlinear Programming Theory, Nonlinear Programming Algorithms, Lecture Notes, 2012, https://www2.isye.gatech.edu/∽nemirovs/OPTIII\_2018\_Transp.pdf
6.
C. Bingham, An antipodally symmetric distribution on the sphere, Ann. Stat., 2 (1974), pp. 1201–1225, https://doi.org/10.1214/aos/1176342874.
7.
S. Bubeck, personal communication, 2021.
8.
S. Bubeck and R. Eldan, The entropic barrier: A simple and optimal universal self-concordant barrier, in Proceedings of The 28th Conference on Learning Theory, Vol. 40 of Proceedings of Machine Learning Research, PMLR, Paris, France, 03-06 Jul 2015, pp. 279–279, http://proceedings.mlr.press/v40/Bubeck15b.html.
9.
S. Chewi, The Entropic Barrier is n-Self-Concordant, preprint, https://arxiv.org/abs/2112.10947 (2021).
10.
M. B. Cohen, A. Madry, D. Tsipras, and A. Vladu, Matrix scaling and balancing via box constrained Newton’s method and interior point methods, in FOCS’17: Proceedings of the 58th Annual IEEE Symposium on Foundations of Computer Science, IEEE, Piscataway, NJ, 2017, pp. 902–913.
11.
T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley, New York, 2012.
12.
W. Dai, Y. Liu, and B. Rider, Quantization bounds on grassmann manifolds and applications to mimo communications, IEEE Trans. Inform. Theory, 54 (2008), pp. 1108–1123.
13.
M. Dudík, S. J. Phillips, and R. E. Schapire, Maximum entropy density estimation with generalized regularization and an application to species distribution modeling, J. Mach. Learn. Res., 8 (2007), pp. 1217–1260.
14.
J. J. Duistermaat and G. J. Heckman, On the variation in the cohomology of the symplectic form of the reduced phase space, Invent. Math., 69 (1982), pp. 259–268, https://doi.org/10.1007/BF01399506.
15.
A. Garg, L. Gurvits, R. Oliveira, and A. Wigderson, Operator scaling: Theory and applications, Found. Comput. Math., (2015), pp. 1–68.
16.
M. X. Goemans and D. P. Williamson, Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming, J. ACM, 42 (1995), pp. 1115–1145, https://doi.org/10.1145/227683.227684.
17.
M. Gromov, Convex sets and Kahler manifolds, in Advances in Differential Geometry and Topology, World Scientific, 1990, pp. 1–38, https://doi.org/10.1142/9789814439381_0001.
18.
O. Güler, Barrier functions in interior point methods, Math. Oper. Res., 21 (1996), pp. 860–885.
19.
O. Güler, On the self-concordance of the universal barrier function, SIAM J. Optimiz., 7 (1997), pp. 295–303, https://doi.org/10.1137/S105262349529180X.
20.
O. Güler and L. Tunçel, Characterization of the barrier parameter of homogeneous convex cones, Math. Program., 81 (1998), pp. 55–76, https://doi.org/10.1007/BF01584844.
21.
L. Gurvits and A. Samorodnitsky, A deterministic polynomial-time algorithm for approximating mixed discriminant and mixed volume, and a combinatorial corollary, Discrete Comput. Geom., 27 (2002), pp. 531–550.
22.
Harish-Chandra, Differential operators on a semisimple Lie algebra, Amer. J. Math., 79 (1957), pp. 87–120, http://www.jstor.org/stable/2372387.
23.
P. D. Hoff, Simulation of the matrix Bingham-von Mises-Fisher distribution, with applications to multivariate and relational data, J. Comput. Graph. Statist., 18 (2009), pp. 438–456, http://www.jstor.org/stable/25651254.
24.
A. Horn, Doubly stochastic matrices and the diagonal of a rotation matrix, Amer. J. Math., 76 (1954), pp. 620–630.
25.
C. Itzykson and J. Zuber, The planar approximation. II, J. Math. Phys., 21 (1980), pp. 411–421, https://doi.org/10.1063/1.524438.
26.
E. T. Jaynes, Information theory and statistical mechanics, Phys. Rev. (2), 106 (1957), pp. 620–630, https://doi.org/10.1103/PhysRev.106.620.
27.
E. T. Jaynes, Information theory and statistical mechanics. II, Phys. Rev. (2), 108 (1957), pp. 171–190, https://doi.org/10.1103/PhysRev.108.171.
28.
E. T. Jaynes, Information theory and statistical mechanics. ii, Phys. Rev., 108 (1957), pp. 171–190, https://doi.org/10.1103/PhysRev.108.171, https://link.aps.org/doi/10.1103/PhysRev.108.171.
29.
C. G. Khatri and K. V. Mardia, The Von Mises-Fisher matrix distribution in orientation statistics, J. Roy. Statist. Soc. Ser. B, 39 (1977), pp. 95–106, http://www.jstor.org/stable/2984884.
30.
B. Klartag, On convex perturbations with a bounded isotropic constant, Geom. Funct. Anal., 16 (2006), pp. 1274–1290, https://doi.org/10.1007/s00039-006-0588-1.
31.
J. Leake, C. McSwiggen, and N. K. Vishnoi, Sampling matrices from Harish-Chandra-Itzykson–Zuber densities with applications to quantum inference and differential privacy, in Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, ACM, New York, 2021, pp. 1384–1397.
32.
J. Leake and N. K. Vishnoi, On the computability of continuous maximum entropy distributions: Adjoint orbits of Lie groups, Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, ACM, New York, 2020, pp. 930–943.
33.
L. Lovász and S. Vempala, Fast algorithms for logconcave functions: Sampling, rounding, integration and optimization, in 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS’06), IEEE Computer Society, Los Alamitos, CA, 2006, pp. 57–68.
34.
Y. Nesterov and A. Nemirovskii, Interior-Point Polynomial Algorithms in Convex Programming, Vol. 13, SIAM, 1994.
35.
Y. Nesterov and A. S. Nemirovskii, Interior-Point Polynomial Algorithms in Convex Programming, SIAM, Philadelphia, 1994, https://doi.org/10.1137/1.9781611970791.
36.
J. L. Park and W. Band, Rigorous information-theoretic derivation of quantum-statistical thermodynamics. I, Found. Phys., 7 (1977), pp. 233–244, https://doi.org/10.1007/BF00709009.
37.
D. Pinasco, Lower bounds for norms of products of polynomials via Bombieri inequality, Trans. Amer. Math. Soc., 364 (2012), pp. 3993–4010.
38.
J. Renegar, A Mathematical View of Interior-Point Methods in Convex Optimization, SIAM, Philadelphia, 2001.
39.
I. Schur, Uber eine klasse von mittelbildungen mit anwendungen auf die determinantentheorie, Sitzungsberichte der Berliner Mathematischen Gesellschaft, 22 (1923), p. 9–29.
40.
M. Singh and N. K. Vishnoi. Entropy, optimization and counting, in Proceedings of the 46th Annual ACM Symposium on Theory of Computing, ACM, New York, 2014, pp. 50–59.
41.
P. B. Slater, Relations between the barycentric and Von Neumann entropies of a density matrix, Phys. Lett. A, 159 (1991), pp. 411–414, https://doi.org/10.1016/0375-9601(91)90371-E.
42.
P. B. Slater, Minimum relative entropies of low-dimensional spin systems, Phys. A, 182 (1992), pp. 145–154.
43.
D. Straszak and N. K. Vishnoi, Maximum entropy distributions: Bit complexity and stability, Proc. Mach. Learn. Res. (PMLR), 99 (2019), pp. 2861–2891, http://proceedings.mlr.press/v99/straszak19a.html.
44.
M. Vergne, Convex polytopes and quantization of symplectic manifolds, Proc. Natl. Acad. Sci. USA, 93 (1996), pp. 14238–14242, https://doi.org/10.1073/pnas.93.25.14238.
45.
R. Vershynin, Introduction to the Non-asymptotic Analysis of Random Matrices, preprint, arXiv:1011.3027, 2010.
46.
N. K. Vishnoi, Algorithms for Convex Optimization, Cambridge University Press, Cambridge, 2021, https://doi.org/10.1017/9781108699211.
47.
J. von Neumann and R. Beyer, Mathematical Foundations of Quantum Mechanics, Goldstine Printed Materials, Princeton University Press, 1955.
48.
M. H. Wright, The interior-point revolution in optimization: History, recent developments, and lasting consequences, Bull. Amer. Math. Soc., 42 (2005), pp. 39–56.
Information & Authors
Information
Published In
Copyright
© 2022 Society for Industrial and Applied Mathematics.
History
Submitted: 17 August 2021
Accepted: 13 June 2022
Published online: 28 October 2022
Keywords
MSC codes
Authors
Funding Information
National Science Foundation (NSF): CCF-1908347
This research was partially supported by NSF CCF-1908347 grant and by Vetenskapsrådet.
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
There are no citations for this item
View Options
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].