Abstract

We study convex relaxations of the image labeling problem on a continuous domain with regularizers based on metric interaction potentials. The generic framework ensures existence of minimizers and covers a wide range of relaxations of the original combinatorial problem. We focus on two specific relaxations that differ in flexibility and simplicity—one can be used to tightly relax any metric interaction potential, while the other covers only Euclidean metrics but requires less computational effort. For solving the nonsmooth discretized problem, we propose a globally convergent Douglas–Rachford scheme and show that a sequence of dual iterates can be recovered in order to provide a posteriori optimality bounds. In a quantitative comparison to two other first-order methods, the approach shows competitive performance on synthetic and real-world images. By combining the method with an improved rounding technique for nonstandard potentials, we were able to routinely recover integral solutions within $1\%$–$5\%$ of the global optimum for the combinatorial image labeling problem.

MSC codes

  1. 90C25
  2. 90C27
  3. 65D18
  4. 68U10
  5. 65K10
  6. 49M20

Keywords

  1. segmentation
  2. continuous cut
  3. convex relaxation
  4. total variation
  5. variational methods
  6. saddle point problem
  7. splitting methods
  8. nonsmooth optimization

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
L. Ambrosio, N. Fusco, and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems, The Clarendon Press, Oxford University Press, New York, 2000.
2.
B. Appleton and H. Talbot, Globally minimal surfaces by continuous maximal flows, IEEE Trans. Pattern Anal. Mach. Intell., 28 (2006), pp. 106–118.
3.
K. J. Arrow, L. Hurwicz, and H. Uzawa, Studies in Linear and Non-Linear Programming, Stanford University Press, Stanford, CA, 1958.
4.
J.-F. Aujol, Some first-order algorithms for total variation based image restoration, J. Math. Imaging Vision, 34 (2009), pp. 307–327.
5.
E. Bae and X.-C. Tai, Graph cut optimization for the piecewise constant level set method applied to multiphase image segmentation, in Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Comput. Sci. 5567, Springer-Verlag, Berlin, 2009, pp. 1–13.
6.
E. Bae, J. Yuan, and X.-C. Tai, Global minimization for continuous multiphase partitioning problems using a dual approach, Int. J. Comput. Vision, 92 (2011), pp. 112–129.
7.
S. Becker, J. Bobin, and E. J. Candès, NESTA: A fast and accurate first-order method for sparse recovery, SIAM J. Imaging Sci., 4 (2011), pp. 1–39.
8.
B. Berkels, An unconstrained multiphase thresholding approach for image segmentation, in Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Comput. Sci. 5567, Springer-Verlag, Berlin, 2009, pp. 26–37.
9.
I. Borg and P. J. F. Groenen, Modern Multidimensional Scaling. Theory and Applications, 2nd ed., Springer, New York, 2005.
10.
Y. Boykov, Computing geodesics and minimal surfaces via graph cuts, in Proceedings of the Ninth IEEE International Conference on Computer Vision, 2003, pp. 26–33.
11.
Y. Boykov and V. Kolmogorov, An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision, IEEE Trans. Pattern Anal. Mach. Intell., 26 (2004), pp. 1124–1137.
12.
Y. Boykov, O. Veksler, and R. Zabih, Fast approximate energy minimization via graph cuts, IEEE Trans. Pattern Anal. Mach. Intell., 23 (2001), pp. 1222–1239.
13.
J. P. Boyle and R. L. Dykstra, A method for finding projection onto the intersection of convex sets in Hilbert spaces, in Advances in Order Restricted Statistical Inference, Lecture Notes in Statist. 37, Springer, Berlin, 1986, pp. 28–47.
14.
A. Braides, $\Gamma$-convergence for Beginners, Oxford University Press, Oxford, UK, 2002.
15.
A. Chambolle, D. Cremers, and T. Pock, A Convex Approach for Computing Minimal Partitions, Technical report 649, CMAP, Ecole Polytechnique, Palaiseau, France, 2008.
16.
A. Chambolle and J. Darbon, On total variation minimization and surface evolution using parametric maximum flows, Int. J. Comput. Vision, 84 (2009), pp. 288–307.
17.
A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), pp. 120–145.
18.
T. F. Chan, S. Esedog¯lu, and M. Nikolova, Algorithms for finding global minimizers of image segmentation and denoising models, J. Appl. Math., 66 (2006), pp. 1632–1648.
19.
P. L. Combettes and J.-C. Pesquet, A proximal decomposition method for solving convex variational inverse problems, Inverse Problems, 24 (2008), 065014.
20.
A. Delaunoy, K. Fundana, E. Prados, and A. Heyden, Convex multi-region segmentation on manifolds, in Proceedings of the 12th IEEE International Conference on Computer Vision, 2009, pp. 662–669.
21.
J. Douglas and H. H. Rachford, On the numerical solution of heat conduction problems in two and three space variables, Trans. Amer. Math. Soc., 82 (1956), pp. 421–439.
22.
V. Duval, J.-F. Aujol, and L. Vese, A projected gradient algorithm for color image decomposition, CMLA Preprint 2008-21, ENS Cachan, Cachan, France, 2008.
23.
J. Eckstein, Splitting Methods for Monotone Operators with Application to Parallel Optimization, Ph.D. thesis, MIT, Cambridge, MA, 1989.
24.
J. Eckstein and D. P. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators, Math. Programming, 55 (1992), pp. 293–318.
25.
W. H. Fleming and R. Rishel, An integral formula for total gradient variation, Arch. Math. (Basel), 11 (1960), pp. 218–222.
26.
N. Gaffke and R. Mathar, A cyclic projection algorithm via duality, Metrika, 36 (1989), pp. 29–54.
27.
D. Goldfarb and S. Ma, Fast Multiple Splitting Algorithms for Convex Optimization, preprint, 2009; available online from http://www.arxiv.org/abs/0912.4570.
28.
A. A. Goldstein, Convex programming in Hilbert space, Bull. Amer. Math. Soc., 70 (1964), pp. 709–710.
29.
T. Goldstein, X. Bresson, and S. Osher, Geometric Applications of the Split Bregman Method: Segmentation and Surface Reconstruction, CAM Report 09-06, UCLA, Los Angeles, CA, 2009.
30.
J.C. Gower, Properties of Euclidean and non-Euclidean distance matrices, Linear Algebra Appl., 67 (1985), pp. 81–97.
31.
A. Graham, Kronecker Products and Matrix Calculus with Applications, Wiley, New York, 1981.
32.
H. Ishikawa, Exact optimization for Markov random fields with convex priors, IEEE Trans. Pattern Anal. Mach. Intell., 25 (2003), pp. 1333–1336.
33.
C. R. Johnson and P. Tarazaga, Connections between the real positive semidefinite and distance matrix completion problems, Linear Algebra Appl., 223/224 (1995), pp. 375–391.
34.
J. M. Kleinberg and E. Tardos, Approximation algorithms for classification problems with pairwise relationships: Metric labeling and Markov random fields, in Proceedings of the 40th Annual IEEE Symposium on Foundations of Computer Science, 1999, pp. 14–23.
35.
K. Kolev, M. Klodt, T. Brox, and D. Cremers, Continuous global optimization in multiview 3D reconstruction, Int. J. Comput. Vision, 84 (2009), pp. 80–96.
36.
V. Kolmogorov and Y. Boykov, What metrics can be approximated by geo-cuts, or global optimization of length/area and flux, in Proceedings of the Tenth IEEE International Conference on Computer Vision, 2005, pp. 564–571.
37.
N. Komodakis and G. Tziritas, Approximate labeling via graph cuts based on linear programming, IEEE Trans. Pattern Anal. Mach. Intell., 29 (2007), pp. 1436–1453.
38.
J. Lellmann, F. Becker, and C. Schnörr, Convex optimization for multi-class image labeling with a novel family of total variation based regularizers, in Proceedings of the 12th IEEE International Conference on Computer Vision, 2009, pp. 646–653.
39.
J. Lellmann, D. Breitenreicher, and C. Schnörr, Fast and exact primal-dual iterations for variational problems in computer vision, in Proceedings of the 11th European Conference on Computer Vision, Lecture Notes in Comput. Sci. 6312, Springer-Verlag, Berlin, 2010, pp. 494–505.
40.
J. Lellmann, J. Kappes, J. Yuan, F. Becker, and C. Schnörr, Convex multi-class image labeling by simplex-constrained total variation, in Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Comput. Sci. 5567, Springer-Verlag, Berlin, 2009, pp. 150–162.
41.
J. Lellmann, F. Lenzen, and C. Schnörr, Optimality bounds for a variational relaxation of the image partitioning problem, in Proceedings of the 8th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer-Verlag, Berlin, 2011, pp. 132–146.
42.
E. S. Levitin and B. T. Polyak, Constrained minimization problems, U.S.S.R. Comput. Math. Math. Phys., 6 (1966), pp. 1–50.
43.
J. Lie, M. Lysaker, and X.-C. Tai, A variant of the level set method and applications to image segmentation, Math. Comp., 75 (2006), pp. 1155–1174.
44.
P. L. Lions and B. Mercier, Splitting algorithms for the sum of two nonlinear operators, SIAM J. Numer. Anal., 16 (1979), pp. 964–979.
45.
C. Michelot, A finite algorithm for finding the projection of a point onto the canonical simplex of $\mathbbm{R}^n$, J. Optim. Theory Appl., 50 (1986), pp. 195–200.
46.
D. Mumford and J. Shah, Optimal approximations by piecewise smooth functions and associated variational problems, Comm. Pure Appl. Math., 42 (1989), pp. 577–685.
47.
K. Murota, Discrete Convex Analysis, Monogr. Discrete Math. Appl. 10, SIAM, Philadelphia, 2003.
48.
Y. Nesterov, Smooth minimization of non-smooth functions, Math. Program, 103 (2005), pp. 127–152.
49.
C. Nieuwenhuis, E. Töppe, and D. Cremers, Space-varying color distributions for interactive multiregion segmentation: Discrete versus continuous approaches, in Proceedings of the 8th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition, 2011, pp. 177–190.
50.
C. Olsson, Global Optimization in Computer Vision: Convexity, Cuts and Approximation Algorithms, Ph.D. thesis, Faculty of Engineering, Centre for Mathematical Sciences, Lund University, Lund, Sweden, 2009.
51.
N. Paragios, Y. Chen, and O. Faugeras, eds., Handbook of Mathematical Models in Computer Vision, Springer, New York, 2006.
52.
T. Pock, A. Chambolle, D. Cremers, and H. Bischof, A convex relaxation approach for computing minimal partitions, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 810–817.
53.
T. Pock, D. Cremers, H. Bischof, and A. Chambolle, An algorithm for minimizing the Mumford-Shah functional, in Proceedings of the 12th IEEE International Conference on Computer Vision, 2009, pp. 1133–1140.
54.
T. Pock, D. Cremers, H. Bischof, and A. Chambolle, Global solutions of variational models with convex regularization, SIAM J. Imaging Sci., 3 (2010), pp. 1122–1145.
55.
L. D. Popov, A modification of the Arrow-Hurwicz method of search for saddle points, Math. Notes, 28 (1980), pp. 845–848.
56.
R. T. Rockafellar, Convex Analysis, Princeton University Press, Princeton, NJ, 1970.
57.
R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, 2nd ed., Springer, Berlin, 2004.
58.
L. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D, 60 (1992), pp. 259–268.
59.
G. Sapiro and D. L. Ringach, Anisotropic diffusion of multi-valued images with applications to color filtering, IEEE Trans. Image Process., 5 (1996), pp. 1582–1586.
60.
S. Setzer, Split Bregman algorithm, Douglas-Rachford splitting and frame shrinkage, in Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Comput. Sci. 5567, Springer-Verlag, Berlin, 2009, pp. 464–476.
61.
S. Setzer, Splitting Methods in Image Processing, Ph.D. thesis, Department of Mathematics and Computer Science, University of Mannheim, Mannheim, Germany, 2009.
62.
G. Strang, Maximal flow through a domain, Math. Programming, 26 (1983), pp. 123–143.
63.
G. Strang, The discrete cosine transform, SIAM Rev., 41 (1999), pp. 135–147.
64.
R. Szeliski, R. Zabih, D. Scharstein, O. Veksler, V. Kolmogorov, A. Agarwala, M. Tappen, and C. Rother, A comparative study of energy minimization methods for Markov random fields, in Proceedings of the 9th European Conference on Computer Vision, Lecture Notes in Comput. Sci. 3952, Springer-Verlag, Berlin, 2006, pp. 19–26.
65.
W. Trobin, T. Pock, D. Cremers, and H. Bischof, Continuous energy minimization via repeated binary fusion, in Proceedings of the 10th European Conference on Computer Vision, Lecture Notes in Comput. Sci. 5305, Springer-Verlag, Berlin, 2008, pp. 677–690.
66.
C. Wang and N. Xiu, Convergence of the gradient projection method for generalized convex minimization, Comput. Optim. Appl., 16 (2000), pp. 111–120.
67.
P. Weiss, L. Blanc-Féraud, and G. Aubert, Efficient schemes for total variation minimization under constraints in image processing, SIAM J. Sci. Comput., 31 (2009), pp. 2047–2080.
68.
M. Werlberger, T. Pock, M. Unger, and H. Bischof, A variational model for interactive shape prior segmentation and real-time tracking, in Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Comput. Sci. 5567, Springer-Verlag, Berlin, 2009, pp. 200–211.
69.
G. Winkler, Image Analysis, Random Fields and Markov Chain Monte Carlo Methods, Springer, Berlin, 2006.
70.
H. Wolkowicz, R. Saigal, and L. Vandenberghe, eds., Handbook of Semidefinite Programming. Theory, Algorithms, and Applications, Kluwer Academic Publishers, Boston, 2000.
71.
S. Xu, Estimation of the convergence rate of Dykstra's cyclic projections algorithm in polyhedral case, Acta Math. Appl. Sinica (English Ser.), 16 (2000), pp. 217–220.
72.
C. Zach, D. Gallup, J.-M. Frahm, and M. Niethammer, Fast global labeling for real-time stereo using multiple plane sweeps, in Proceedings of the Vision, Modeling, and Visualization Conference 2008, Konstanz, Germany, Aka GmbH, Heidelberg, 2008, pp. 243–252.
73.
C. Zach, M. Niethammer, and J.-M. Frahm, Continuous maximal flows and Wulff shapes: Application to MRFs, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 1911–1918.
74.
M. Zhu and T. Chan, An Efficient Primal-Dual Hybrid Gradient Algorithm for Total Variation Image Restoration, CAM Report 08-34, UCLA, Los Angeles, CA, 2008.
75.
W. P. Ziemer, Weakly Differentiable Functions. Sobolev Spaces and Functions of Bounded Variation, Springer, New York, 1989.

Information & Authors

Information

Published In

cover image SIAM Journal on Imaging Sciences
SIAM Journal on Imaging Sciences
Pages: 1049 - 1096
ISSN (online): 1936-4954

History

Submitted: 19 August 2010
Accepted: 8 September 2011
Published online: 22 November 2011

MSC codes

  1. 90C25
  2. 90C27
  3. 65D18
  4. 68U10
  5. 65K10
  6. 49M20

Keywords

  1. segmentation
  2. continuous cut
  3. convex relaxation
  4. total variation
  5. variational methods
  6. saddle point problem
  7. splitting methods
  8. nonsmooth optimization

Authors

Affiliations

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media

The SIAM Publications Library now uses SIAM Single Sign-On for individuals. If you do not have existing SIAM credentials, create your SIAM account https://my.siam.org.