Abstract

Tensor recovery has recently arisen in a lot of application fields, such as transportation, medical imaging, and remote sensing. Under the assumption that signals possess sparse and/or low-rank structures, many tensor recovery methods have been developed to apply various regularization techniques together with the operator-splitting type of algorithms. Due to the unprecedented growth of data, it becomes increasingly desirable to use streamlined algorithms to achieve real-time computation, such as stochastic optimization algorithms that have recently emerged as an efficient family of methods in machine learning. In this work, we propose a novel algorithmic framework based on the Kaczmarz algorithm for tensor recovery. We provide thorough convergence analysis and its applications from the vector case to the tensor one. Numerical results on a variety of tensor recovery applications, including sparse signal recovery, low-rank tensor recovery, image inpainting, and deconvolution, illustrate the enormous potential of the proposed methods.

Keywords

  1. Kaczmarz algorithm
  2. tensor recovery
  3. image inpainting
  4. image deblurring
  5. randomized algorithm

MSC codes

  1. 68Q25
  2. 68R10
  3. 68U05

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
M. A. Abidi, A. V. Gribok, and J. Paik, Optimization Techniques in Computer Vision, Springer, New York, 2016.
2.
A. Agaskar, C. Wang, and Y. M. Lu, Randomized Kaczmarz algorithms: Exact MSE analysis and optimal sampling probabilities, in Proceedings of the 2014 IEEE Global Conference on Signal and Information Processing, IEEE, 2014, pp. 389--393.
3.
A. H. Andersen and A. C. Kak, Simultaneous algebraic reconstruction technique (SART): A superior implementation of the ART algorithm, Ultrasonic Imaging, 6 (1984), pp. 81--94.
4.
Z.-Z. Bai and W.-T. Wu, On greedy randomized Kaczmarz method for solving large sparse linear systems, SIAM J. Sci. Comput., 40 (2018), pp. A592--A606.
5.
H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, CMS Books Math. 408, Springer, New York, 2011.
6.
S. Boyd, N. Parikh, and E. Chu, Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, Now Publishers, Boston, 2011.
7.
A. Buades, B. Coll, and J.-M. Morel, Non-local means denoising, IPOL J. Image Process. Online, 1 (2011), pp. 208--212.
8.
J.-F. Cai, E. J. Candès, and Z. Shen, A singular value thresholding algorithm for matrix completion, SIAM J. Optim., 20 (2010), pp. 1956--1982.
9.
J.-F. Cai, S. Osher, and Z. Shen, Convergence of the linearized Bregman iteration for $\ell_1$-norm minimization, Math. Comp., 78 (2009), pp. 2127--2136.
10.
Y. Censor and S. A. Zenios, Parallel Optimization: Theory, Algorithms, and Applications, Oxford University Press, New York, 1997.
11.
S. H. Chan, R. Khoshabeh, K. B. Gibson, P. E. Gill, and T. Q. Nguyen, An augmented Lagrangian method for total variation video restoration, IEEE Trans. Image Processing, 20 (2011), pp. 3097--3111.
12.
S. H. Chan, X. Wang, and O. A. Elgendy, Plug-and-play ADMM for image restoration: Fixed-point convergence and applications, IEEE Trans. Comput. Imaging, 3 (2016), pp. 84--98.
13.
X. Chen and A. M. Powell, Almost sure convergence of the Kaczmarz algorithm with random measurements, J. Fourier Anal. Appl., 18 (2012), pp. 1195--1214.
14.
X. Chen and A. M. Powell, Randomized subspace actions and fusion frames, Constr. Approx., 43 (2016), pp. 103--134.
15.
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Image denoising with block-matching and 3D filtering, in Image Processing: Algorithms and Systems, Neural Networks, and Machine Learning, Proc. SPIE, 6064, International Society for Optics and Photonics, 2006, 606414.
16.
L. Dai, M. Soltanalian, and K. Pelckmans, On the randomized Kaczmarz algorithm, IEEE Signal Process. Lett., 21 (2013), pp. 330--333.
17.
N. Durgin, R. Grotheer, C. Huang, S. Li, A. Ma, D. Needell, and J. Qin, Sparse Randomized Kaczmarz for Support Recovery of Jointly Sparse Corrupted Multiple Measurement Vectors, in Research in Data Science, Springer, New York, 2019, pp. 1--14.
18.
Y. C. Eldar and D. Needell, Acceleration of randomized Kaczmarz method via the Johnson--Lindenstrauss lemma, Numer. Algorithms, 58 (2011), pp. 163--177.
19.
H. Fan, Y. Chen, Y. Guo, H. Zhang, and G. Kuang, Hyperspectral image restoration using low-rank tensor recovery, IEEE J. Selected Topics Applied Earth Observations Remote Sensing, 10 (2017), pp. 4589--4604.
20.
P. Getreuer, Total variation inpainting using split Bregman, IPOL J. Image Process. Online, 2 (2012), pp. 147--157.
21.
R. Gordon, R. Bender, and G. T. Herman, Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography, J. Theoret. Biol., 29 (1970), pp. 471--481.
22.
G. T. Herman and L. B. Meyer, Algebraic reconstruction techniques can be made computationally efficient (positron emission tomography application), IEEE Trans. Medical Imaging, 12 (1993), pp. 600--609.
23.
H. Jeong and C. S. Güntürk, Convergence of the Randomized Kaczmarz Method for Phase Retrieval, preprint, arXiv:1706.10291, 2017.
24.
S. Kaczmarz, Angenaherte auflosung von systemen linearer glei-chungen, Bull. Int. Acad. Pol. Sic. Let., Cl. Sci. Math. Nat., 35 (1937), pp. 355--357.
25.
M. E. Kilmer, K. Braman, N. Hao, and R. C. Hoover, Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging, SIAM J. Matrix Anal. Appl., 34 (2013), pp. 148--172.
26.
M. E. Kilmer and C. D. Martin, Factorization strategies for third-order tensors, Linear Algebra Appl., 435 (2011), pp. 641--658.
27.
M. E. Kilmer, C. D. Martin, and L. Perrone, A Third-Order Generalization of the Matrix SVD as a Product of Third-Order Tensors, Tech. rep. TR-2008-4, Department of Computer Science, Tufts University, 2008.
28.
S. Kindermann and A. Leitao, Convergence rates for Kaczmarz-type regularization methods, Inverse Problems Imaging, 8 (2014).
29.
M.-J. Lai and W. Yin, Augmented $\ell_1$ and nuclear-norm models with a globally linearly convergent algorithm, SIAM J. Imaging Sci., 6 (2013), pp. 1059--1091.
30.
T. Li, T.-J. Kao, D. Isaacson, J. C. Newell, and G. J. Saulnier, Adaptive Kaczmarz method for image reconstruction in electrical impedance tomography, Physiological Measurement, 34 (2013), p. 595.
31.
J. Liu, S. J. Wright, and S. Sridhar, An Asynchronous Parallel Randomized Kaczmarz Algorithm, preprint, arXiv:1401.4780, 2014.
32.
D. A. Lorenz, F. Schöpfer, and S. Wenger, The linearized Bregman method via split feasibility problems: Analysis and generalizations, SIAM J. Imaging Sci., 7 (2014), pp. 1237--1262.
33.
C. Lu, J. Feng, Y. Chen, W. Liu, Z. Lin, and S. Yan, Tensor robust principal component analysis with a new tensor nuclear norm, IEEE Trans. Pattern Anal. Machine Intelligence, 42 (2019), pp. 925--938.
34.
A. Ma and D. Molitor, Randomized Kaczmarz for Tensor Linear Systems, preprint, arXiv:2006.01246, 2020.
35.
A. Ma, D. Needell, and A. Ramdas, Convergence properties of the randomized extended Gauss--Seidel and Kaczmarz methods, SIAM J. Matrix Anal. Appl., 36 (2015), pp. 1590--1604.
36.
D. Needell, Randomized Kaczmarz solver for noisy linear systems, BIT, 50 (2010), pp. 395--403.
37.
D. Needell, N. Srebro, and R. Ward, Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm, Math. Program., 155 (2016), pp. 549--573.
38.
D. Needell and J. A. Tropp, Paved with good intentions: Analysis of a randomized block Kaczmarz method, Linear Algebra Appl., 441 (2014), pp. 199--221.
39.
D. Needell, R. Zhao, and A. Zouzias, Randomized block Kaczmarz method with projection for solving least squares, Linear Algebra Appl., 484 (2015), pp. 322--343.
40.
J. E. Peterson, B. N. Paulsson, and T. V. McEvilly, Applications of algebraic reconstruction techniques to crosshole seismic data, Geophysics, 50 (1985), pp. 1566--1580.
41.
S. Petra, Randomized sparse block Kaczmarz as randomized dual block-coordinate descent, An. Ştiinţ. Univ. ``Ovidius" Constanţa Ser. Mat., 23 (2015), pp. 129--149.
42.
C. Popa and R. Zdunek, Kaczmarz extended algorithm for tomographic image reconstruction from limited-data, Math. Comput. Simul., 65 (2004), pp. 579--598.
43.
R. T. Rockafellar, Convex Analysis, Princeton Math. Ser. 28, Princeton University Press, Princeton, NJ, 1970.
44.
F. Schöpfer, Linear convergence of descent methods for the unconstrained minimization of restricted strongly convex functions, SIAM J. Optim., 26 (2016), pp. 1883--1911.
45.
F. Schöpfer and D. A. Lorenz, Linear convergence of the randomized sparse Kaczmarz method, Math. Program., 173 (2019), pp. 509--536.
46.
O. Semerci, N. Hao, M. E. Kilmer, and E. L. Miller, Tensor-based formulation and nuclear norm regularization for multienergy computed tomography, IEEE Trans. Image Process., 23 (2014), pp. 1678--1693.
47.
J. Shen and T. F. Chan, Mathematical models for local nontexture inpaintings, SIAM J. Appl. Math., 62 (2002), pp. 1019--1043.
48.
T. Strohmer and R. Vershynin, A randomized Kaczmarz algorithm with exponential convergence, J. Fourier Anal. Appl., 15 (2009), 262.
49.
H. Tan, G. Feng, J. Feng, W. Wang, Y.-J. Zhang, and F. Li, A tensor-based method for missing traffic data completion, Transportation Research Part C Emerging Technologies, 28 (2013), pp. 15--27.
50.
Y. S. Tan and R. Vershynin, Phase retrieval via randomized Kaczmarz: Theoretical guarantees, Inform. Inference, 8 (2018), pp. 97--123.
51.
S. J. Wright, Coordinate descent algorithms, Math. Program., 151 (2015), pp. 3--34.
52.
W. Yin, S. Osher, D. Goldfarb, and J. Darbon, Bregman iterative algorithms for $\ell_1$-minimization with applications to compressed sensing, SIAM J. Imaging Sci., 1 (2008), pp. 143--168.
53.
H. Zhou, L. Li, and H. Zhu, Tensor regression with applications in neuroimaging data analysis, J. Amer. Statist. Assoc., 108 (2013), pp. 540--552.
54.
A. Zouzias and N. M. Freris, Randomized extended Kaczmarz for solving least squares, SIAM J. Matrix Anal. Appl., 34 (2013), pp. 773--793.

Information & Authors

Information

Published In

cover image SIAM Journal on Imaging Sciences
SIAM Journal on Imaging Sciences
Pages: 1439 - 1471
ISSN (online): 1936-4954

History

Submitted: 12 February 2021
Accepted: 13 July 2021
Published online: 18 October 2021

Keywords

  1. Kaczmarz algorithm
  2. tensor recovery
  3. image inpainting
  4. image deblurring
  5. randomized algorithm

MSC codes

  1. 68Q25
  2. 68R10
  3. 68U05

Authors

Affiliations

Funding Information

National Science Foundation https://doi.org/10.13039/100000001 : DMS-2050028, DMS-1941197

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

View Options

View options

PDF

View PDF

Figures

Tables

Media

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media