Methods and Algorithms for Scientific Computing

A New Truncation Strategy for the Higher-Order Singular Value Decomposition

Abstract

We present an alternative strategy for truncating the higher-order singular value decomposition (T-HOSVD). An error expression for an approximate Tucker decomposition with orthogonal factor matrices is presented, leading us to propose a novel truncation strategy for the HOSVD, which we refer to as the sequentially truncated higher-order singular value decomposition (ST-HOSVD). This decomposition retains several favorable properties of the T-HOSVD, while reducing the number of operations required to compute the decomposition and practically always improving the approximation error. Three applications are presented, demonstrating the effectiveness of ST-HOSVD. In the first application, ST-HOSVD, T-HOSVD, and higher-order orthogonal iteration (HOOI) are employed to compress a database of images of faces. On average, the ST-HOSVD approximation was only $0.1\%$ worse than the optimum computed by HOOI, while cutting the execution time by a factor of $20$. In the second application, classification of handwritten digits, ST-HOSVD achieved a speedup factor of $50$ over T-HOSVD during the training phase, and reduced the classification time and storage costs, while not significantly affecting the classification error. The third application demonstrates the effectiveness of ST-HOSVD in compressing results from a numerical simulation of a partial differential equation. In such problems, ST-HOSVD inevitably can greatly improve the running time. We present an example wherein the 2 hour $45$ minute calculation of T-HOSVD was reduced to just over one minute by ST-HOSVD, representing a speedup factor of $133$, while even improving the memory consumption.

MSC codes

  1. 15A03
  2. 15A69
  3. 15A72
  4. 65F99
  5. 65Y20

Keywords

  1. tensor
  2. sequentially truncated higher-order singular value decomposition
  3. higher-order singular value decomposition
  4. multilinear singular value decomposition
  5. multilinear orthogonal projection

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorensen, LAPACK Users' Guide, Third Edition, SIAM, Philadelphia, PA, 1999.
2.
C. Andersson and R. Bro, Improving the speed of multi-way algorithms: Part I. Tucker3, Chemometrics and Intelligent Laboratory Systems, 42 (1998), pp. 93–103.
3.
B. Bader and T. Kolda, Algorithm $862$: MATLAB tensor classes for fast algorithm prototyping, ACM Trans. Math. Software, 32 (2006), pp. 635–653.
4.
B. Bader and T. Kolda, MATLAB Tensor Toolbox Version $2.4$, Mar. 2010, current version available online at http://www.sandia.gov/~tgkdda/TensorToolbox/.
5.
F. Bell, D. Lambrecht, and M. Head-Gordon, Higher order singular value decomposition in quantum chemistry, Molec. Phys., 108 (2010), pp. 2759–2773.
6.
R. Bro and C. Andersson, Improving the speed of multiway algorithms: Part II: Compression, Chemometrics and Intelligent Laboratory Systems, 42 (1998), pp. 105–113.
7.
J. Chen and Y. Saad, On the tensor SVD and the optimal low rank orthogonal approximation of tensors, SIAM J. Matrix Anal. Appl., 30 (2009), pp. 1709–1734.
8.
P. Comon, G. Golub, L.-H. Lim, and B. Mourrain, Symmetric tensors and symmetric tensor rank, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1254–1279.
9.
R. Costantini, L. Sbaiz, and S. Süsstrunk, Higher order SVD analysis for dynamic texture synthesis, IEEE Trans. Image Process., 17 (2008), pp. 42–52.
10.
L. De Lathauwer, B. De Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1253–1278.
11.
L. De Lathauwer, B. De Moor, and J. Vandewalle, On the best rank-1 and rank-$(R_1$, $R_2$, $R_n)$ approximation of higher-order tensors, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1324–1342.
12.
L. De Lathauwer and J. Vandewalle, Dimensionality reduction in higher-order signal processing and rank-$(r_1,r_2,\ldots,r_n)$ reduction in multilinear algebra, Linear Algebra Appl., 391 (2004), pp. 31–55.
13.
V. de Silva and L.-H. Lim, Tensor rank and the ill-posedness of the best low-rank approximation problem, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1084–1127.
14.
J. W. Demmel, Applied Numerical Linear Algebra, SIAM, Philadelphia, PA, 1997.
15.
L. Eldén and B. Savas, A Newton–Grassmann method for computing the best multilinear rank-$(r_1,$ $r_2,$ $r_3)$ approximation of a tensor, SIAM J. Matrix Anal. Appl., 31 (2009), pp. 248–271.
16.
F. Estienne, N. Matthijs, D. Massart, P. Ricoux, and D. Leibovici, Multi-way modelling of high-dimensionality electroencephalographic data, Chemometrics and Intelligent Laboratory Systems, 58 (2001), pp. 59–72.
17.
G. Golub and C. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996.
18.
L. Grasedyck, Hierarchical singular value decomposition of tensors, SIAM J. Matrix Anal. Appl., 31 (2010), pp. 2029–2054.
19.
M. Haardt, F. Roemer, and G. Del Galdo, Higher-order SVD-based subspace estimation to improve the parameter estimation accuracy in multidimensional harmonic retrieval problems, IEEE Trans. Signal Process., 56 (2008), pp. 3198–3213.
20.
W. Hackbusch and B. Khoromskij, Tensor-product approximation to operators and functions in high dimensions, J. Complexity, 23 (2007), pp. 697–714.
21.
W. Hackbusch and S. Kühn, A new scheme for the tensor representation, J. Fourier Anal. Appl., 15 (2009), pp. 706–722.
22.
F. Hitchcock, Multiple invariants and generalized rank of a p-way matrix or tensor, J. Math. Phys., 7 (1927), pp. 39–79.
23.
M. Ishteva, P.-A. Absil, S. Van Huffel, and L. De Lathauwer, Tucker compression and local optima, Chemometrics and Intelligent Laboratory Systems, 106 (2010), pp. 57–64.
24.
M. Ishteva, P.-A. Absil, S. Van Huffel, and L. De Lathauwer, Best low multilinear rank approximation of higher-order tensors, based on the Riemannian trust-region scheme, SIAM J. Matrix Anal. Appl., 32 (2011), pp. 115–135.
25.
M. Ishteva, L. De Lathauwer, P.-A. Absil, and S. Van Huffel, Dimensionality reduction for higher-order tensors: Algorithms and applications, Int. J. Pure Appl. Math., 42 (2008), pp. 337–343.
26.
M. Ishteva, L. De Lathauwer, P.-A. Absil, and S. Van Huffel, Differential-geometric Newton method for the best rank-$(r_1, r_2, r_3)$ approximation of tensors, Numer. Algorithms, 51 (2009), pp. 179–194.
27.
B. Khoromskij, Structured rank-$(r_1, \ldots, r_d)$ decomposition of function-related tensors in $r^d$, Comput. Methods Appl. Math., 6 (2006), pp. 194–220.
28.
E. Kofidis and P. A. Regalia, On the best rank-1 approximation of higher-order supersymmetric tensors, SIAM J. Matrix Anal. Appl., 23 (2002), pp. 863–884.
29.
T. G. Kolda and B. W. Bader, Tensor decompositions and applications, SIAM Rev., 51 (2009), pp. 455–500.
30.
P. Kroonenberg and J. de Leeuw, Principal component analysis of three-mode data by means of alternating least squares algorithms, Psychometrika, 45 (1980), pp. 69–97.
31.
C. Liu, K. Nakashima, H. Sako, and H. Fujisawa, Handwritten digit recognition: Benchmarking of state-of-the-art techniques, Pattern Recognition, 36 (2003), pp. 2271–2285.
32.
C. Liu, J. Zhou, and K. He, Image compression based on truncated HOSVD, in Proceedings of the International Conference on Information Engineering and Computer Science, IEEE Press, Piscataway, NJ, 2009, pp. 1–4.
33.
N. Liu, B. Zhang, J. Yan, Z. Chen, W. Liu, F. Bai, and L. Chien, Text representation: From vector to tensor, Proceedings of the IEEE International Conference on Data Mining, J. Han, B. W. Wah, V. Raghavan, X. Wu, and R. Rastogi, eds., IEEE Computer Society, Los Alamitos, CA, 2005, pp. 725–728.
34.
X. Liu, L. De Lathauwer, F. Janssens, and B. De Moor, Hybrid clustering of multiple information sources via HOSVD, in Advances in Neural Networks - ISNN 2010, L. Zhang, B.-L. Lu, and J. Kwok, eds., Lecture Notes in Comput. Sci. 6064, Springer, Berlin, Heidelberg, 2010, pp. 337–345.
35.
L. Lorente, J. Vega, and A. Velazquez, Compression of aerodynamic databases using high-order singular value decomposition, Aerospace Sci. Tech., 14 (2010), pp. 168–177.
36.
M. Markaki and Y. Stylianou, Discrimination of speech from nonspeech in broadcast news based on modulation frequency features, Speech Communication, 53 (2011), pp. 726–735.
37.
N. Mesgarani, M. Slaney, and S. Shamma, Discrimination of speech from nonspeech based on multiscale spectro-temporal modulations, IEEE Trans. Audio Speech, Language Process., 14 (2006), pp. 920–930.
38.
D. Muti and S. Bourennane, Survey on tensor signal algebraic filtering, Signal Process., 87 (2007), pp. 237–249.
39.
J. Nagy and M. Kilmer, Kronecker product approximation for preconditioning in three-dimensional imaging applications, IEEE Trans. Image Process., 15 (2006), pp. 604–613.
40.
L. Omberg, G. Golub, and O. Alter, A tensor higher-order singular value decomposition for integrative analysis of DNA microarray data from different studies, Proc. Natl. Acad. Sci. USA, 104 (2007), pp. 18371–18376.
41.
L. Omberg, J. Meyerson, K. Kobayashi, L. Drury, J. Diffley, and O. Alter, Global effects of DNA replication and DNA replication origin activity on eukaryotic gene expression, Molec. Systems Biol., 5 (2009), 312.
42.
I. V. Oseledets, Tensor-train decomposition, SIAM J. Sci. Comput., 33 (2011), pp. 2295–2317.
43.
I. V. Oseledets and E. E. Tyrtyshnikov, Breaking the curse of dimensionality, or how to use SVD in many dimensions, SIAM J. Sci. Comput., 31 (2009), pp. 3744–3759.
44.
I. Oseledets and E. Tyrtyshnikov, Recursive decomposition of multidimensional tensors, Dokl. Math., 80 (2009), pp. 460–462.
45.
J. M. Papy, L. De Lathauwer, and S. Van Huffel, Exponential data fitting using multilinear algebra: The single-channel and multi-channel case, Numer. Linear Algebra Appl., 12 (2005), pp. 809–826.
46.
Y. Saad, Numerical Methods for Large Eigenvalue Problems, Manchester University Press, Manchester, UK, 1992.
47.
Y. Saad, Iterative Methods for Sparse Linear Systems, Second Edition, SIAM, Philadelphia, PA, 2003.
48.
F. Samaria and A. Harter, Parameterisation of a stochastic model for human face identification, in Proceedings of the Second IEEE Workshop on Applications of Computer Vision, IEEE Press, Piscataway, NJ, pp. 138–142.
49.
B. Savas, Analyses and Tests of Handwritten Digit Recognition Algorithms, examensarbete (Master's thesis), Department of Mathematics, Tekniska Högskolan i Linköping, Linköping, Sweden, 2003.
50.
B. Savas and L. Eldén, Handwritten digit classification using higher order singular value decomposition, Pattern Recognition, 40 (2007), pp. 993–1003.
51.
B. Savas and L.-H. Lim, Quasi-Newton methods on Grassmannians and multilinear approximations of tensors, SIAM J. Sci. Comput., 32 (2010), pp. 3352–3393.
52.
J.-T. Sun, H.-J. Zeng, H. Liu, Y. Lu, and Z. Chen, CubeSVD: A novel approach to personalized web search, in Proceedings of the 14th international conference on World Wide Web (WWW '05), ACM, New York, 2005, pp. 382–390.
53.
P. Symeonidis, A. Nanopoulos, and Y. Manolopoulos, Tag recommendations based on tensor dimensionality reduction, in Proceedings of the 2008 ACM Conference on Recommender Systems, New York, 2008, ACM, New York, 2008, pp. 43–50.
54.
L. Tucker, Implications of factor analysis of three-way matrices for measurement of change, in Problems in Measuring Change, C. Harris, ed., University of Wisconsin Press, Madison, WI, 1963, pp. 122–137.
55.
L. Tucker, The extension of factor analysis to three-dimensional matrices, in Contributions to Mathematical Psychology, H. Gulliksen and N. Frederiksen, eds., Holt, Rinehardt, & Winston, New York, 1964, pp. 110–127.
56.
L. Tucker, Some mathematical notes on three-mode factor analysis, Psychometrika, 31 (1966), pp. 279–311.
57.
C. Van Loan, The ubiquitous Kronecker product, J. Comput. Appl. Math., 123 (2000), pp. 85–100.
58.
N. Vannieuwenhoven, R. Vandebril, and K. Meerbergen, On the Truncated Multilinear Singular Value Decomposition, Technical Report TW589, Department of Computer Science, Katholieke Universiteit Leuven, Leuven, Belgium, 2011.
59.
M. Vasilescu and D. Terzopoulos, Multilinear image analysis for facial recognition, in Proceedings of the 16th International Conference on Pattern Recognition, IEEE Computer Society, Los Alamitos, CA, 2002, Vol. 2, pp. 511–514.
60.
M. Vasilescu and D. Terzopoulos, Multilinear subspace analysis of image ensembles, IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition, 2 (2003), pp. 93–99.
61.
M. Vasilescu and D. Terzopoulos, TensorTextures: Multilinear image-based rendering, ACM Trans. Graph., 23 (2004), pp. 336–342.
62.
H. Wang and N. Ahuja, Facial expression decomposition, in Proceedings of the Ninth IEEE International Conference on Computer Vision., 2003, Vol. 2, pp. 958–965.
63.
T. Zhang and G. H. Golub, Rank-one approximation to high order tensors, SIAM J. Matrix Anal. Appl., 23 (2001), pp. 534–550.

Information & Authors

Information

Published In

cover image SIAM Journal on Scientific Computing
SIAM Journal on Scientific Computing
Pages: A1027 - A1052
ISSN (online): 1095-7197

History

Submitted: 1 June 2011
Accepted: 26 January 2012
Published online: 3 April 2012

MSC codes

  1. 15A03
  2. 15A69
  3. 15A72
  4. 65F99
  5. 65Y20

Keywords

  1. tensor
  2. sequentially truncated higher-order singular value decomposition
  3. higher-order singular value decomposition
  4. multilinear singular value decomposition
  5. multilinear orthogonal projection

Authors

Affiliations

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media

The SIAM Publications Library now uses SIAM Single Sign-On for individuals. If you do not have existing SIAM credentials, create your SIAM account https://my.siam.org.