An AO-ADMM Approach to Constraining PARAFAC2 on All Modes
Abstract
Analyzing multiway measurements with variations across one mode of the dataset is a challenge in various fields including data mining, neuroscience, and chemometrics. For example, measurements may evolve over time or have unaligned time profiles. The PARAFAC2 model has been successfully used to analyze such data by allowing the underlying factor matrices in one mode (i.e., the evolving mode) to change across slices. The traditional approach to fit a PARAFAC2 model is to use an alternating least squares--based algorithm, which handles the constant cross-product constraint of the PARAFAC2 model by implicitly estimating the evolving factor matrices. This approach makes imposing regularization on these factor matrices challenging. There is currently no algorithm to flexibly impose such regularization with general penalty functions and hard constraints. In order to address this challenge and to avoid the implicit estimation, in this paper, we propose an algorithm for fitting PARAFAC2 based on alternating optimization with the alternating direction method of multipliers (AO-ADMM). With numerical experiments on simulated data, we show that the proposed PARAFAC2 AO-ADMM approach allows for flexible constraints, recovers the underlying patterns accurately, and is computationally efficient compared to the state-of-the-art. We also apply our model to two real-world datasets from neuroscience and chemometrics, and show that constraining the evolving mode improves the interpretability of the extracted patterns.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
Supplementary Material
PLEASE NOTE: These supplementary files have not been peer-reviewed.
Index of Supplementary Materials
Title of paper: An AO-Admm Approach to Constraining PARAFAC2 on All Modes
Authors: Marie Roald, Carla Schenker, Vince D. Calhoun, Tulay Adali, Rasmus Bro, Jeremy E. Cohen, and Evrim Acar
File: supplement.pdf
Type: PDF
Contents: Proofs, more details, and additional simulation experiments.
References
1.
E. Acar, C. A. Bingol, H. Bingol, R. Bro, and B. Yener, Multiway analysis of epilepsy tensors, Bioinformatics, 23 (2007), pp. i10--i18.
2.
E. Acar and B. Yener, Unsupervised multiway data analysis: A literature survey, IEEE Trans. Knowl. Data Eng., 21 (2009), pp. 6--20.
3.
A. Afshar, I. Perros, E. E. Papalexakis, E. Searles, J. Ho, and J. Sun, COPA: Constrained PARAFAC2 for sparse & large datasets, in Proceedings of CIKM'18, ACM, 2018, pp. 793--802.
4.
J. M. Amigo, T. Skov, R. Bro, J. Coello, and S. Maspoch, Solving GC-MS problems with PARAFAC2, TrAC Trends Anal. Chem., 27 (2008), pp. 714--725.
5.
Z. Bai, P. Walker, A. Tschiffely, F. Wang, and I. Davidson, Unsupervised network discovery for brain imaging data, in Proceedings of KDD'17, 2017, pp. 55--64.
6.
A. Beck, First-Order Methods in Optimization, SIAM, Philadelphia, 2017.
7.
D. Bertsekas, Nonlinear Programming, Athena Sci. Optim. Comput. Ser., Athena Scientific, Nashua, NH, 1999.
8.
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Mach. Learn., 3 (2011), pp. 1--122.
9.
S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, Cambridge, UK, 2004.
10.
R. Bro, PARAFAC. tutorial and applications, Chemom. Intel. Lab. Syst., 38 (1997), pp. 149--172.
11.
R. Bro, Multi-way Analysis in the Food Industry, Ph.D. thesis, Royal Veterinary and Agricultural University, Denmark, 1998.
12.
R. Bro and C. A. Andersson, Improving the speed of multiway algorithms: Part II: Compression, Chemom. Intel. Lab. Syst., 42 (1998), pp. 105--113.
13.
R. Bro, C. A. Andersson, and H. A. L. Kiers, PARAFAC2 - Part II. Modeling chromatographic data with retention time shifts, J. Chemom., 13 (1999), pp. 295--309.
14.
R. Bro and S. De Jong, A fast non-negativity-constrained least squares algorithm, J. Chemom., 11 (1997), pp. 393--401.
15.
R. Bro, R. Leardi, and L. G. Johnsen, Solving the sign indeterminacy for multiway models, J. Chemom., 27 (2013), pp. 70--75.
16.
R. Bro and N. D. Sidiropoulos, Least squares algorithms under unimodality and non-negativity constraints, J. Chemom., 12 (1998), pp. 223--247.
17.
J. D. Carroll and J. J. Chang, Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition, Psychometrika, 35 (1970), pp. 283--319.
18.
J. D. Carroll, S. Pruzansky, and J. B. Kruskal, Candelinc: A general approach to multidimensional analysis of many-way arrays with linear constraints on parameters, Psychometrika, 45 (1980), pp. 3--24.
19.
C. Chatzichristos, E. Kofidis, M. Morante, and S. Theodoridis, Blind fMRI source unmixing via higher-order tensor decompositions, J. Neurosci. Methods, 315 (2019), pp. 17--47.
20.
P. A. Chew, B. W. Bader, T. G. Kolda, and A. Abdelali, Cross-language information retrieval using PARAFAC2, in Proceedings of KDD'07, 2007, pp. 143--152.
21.
J. E. Cohen and R. Bro, Nonnegative PARAFAC2: A flexible coupling approach, in Proceedings of LVA/ICA'18, 2018, pp. 89--98.
22.
L. Condat, A direct algorithm for 1-D total variation denoising, IEEE Signal Process. Lett., 20 (2013), pp. 1054--1057.
23.
L. Condat, Software, 2019, https://lcondat.github.io/software.html.
24.
L. De Lathauwer, Decompositions of a higher-order tensor in block terms---part II: Definitions and uniqueness, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1033--1066.
25.
D. M. Dunlavy, T. G. Kolda, and E. Acar, Temporal link prediction using matrix and tensor factorizations, ACM TKDD, 5 (2011), 10.
26.
S. Friedland and G. Ottaviani, The number of singular vector tuples and uniqueness of best rank-one approximation of tensors, Found. Comput. Math., 14 (2014), pp. 1209--1242.
27.
M. P. Friedlander and K. Hatz, Computing non-negative tensor factorizations, Optim. Methods Softw., 23 (2008), pp. 631--647.
28.
N. Gillis and F. Glineur, Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization, Neural Comput., 24 (2012), pp. 1085--1105.
29.
R. L. Gollub et al., The MCIC collection: A shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia, Neuroinformatics, 11 (2013), pp. 367--388.
30.
E. Gujral, R. Pasricha, and E. Papalexakis, Beyond rank-1: Discovering rich community structure in multi-aspect graphs, in Proceedings of WWW'20, 2020, pp. 452--462.
31.
R. A. Harshman, Foundations of the PARAFAC Procedure: Models and Conditions for an “Explanatory” Multi-Modal Factor Analysis, UCLA Working Papers in Phonetics 16, UCLA, Los Angeles, CA, 1970, pp. 1--84.
32.
R. A. Harshman, PARAFAC2: Mathematical and Technical Notes, UCLA Working Papers in Phonetics 22, UCLA, Los Angeles, CA, 1972, pp. 30--44.
33.
R. A. Harshman and M. E. Lundy, Uniqueness proof for a family of models sharing features of Tucker's three-mode factor analysis and parafac/candecomp, Psychometrika, 61 (1996), pp. 133--154.
34.
N. E. Helwig, The special sign indeterminacy of the direct-fitting Parafac2 model: Some implications, cautions, and recommendations for simultaneous component analysis, Psychometrika, 78 (2013), pp. 725--739.
35.
N. E. Helwig, Estimating latent trends in multivariate longitudinal data via Parafac2 with functional and structural constraints, Biomed. J., 59 (2017), pp. 783--803.
36.
F. L. Hitchcock, The expression of a tensor or a polyadic as a sum of products, J. Math. Phys., 6 (1927), pp. 164--189.
37.
K. Huang, N. D. Sidiropoulos, and A. P. Liavas, A flexible and efficient algorithmic framework for constrained matrix and tensor factorization, IEEE Trans. Signal Process., 64 (2016), pp. 5052--5065.
38.
H. A. L. Kiers, A three-step algorithm for CANDECOMP/PARAFAC analysis of large data sets with multicollinearity, J. Chemom., 12 (1998), pp. 155--171.
39.
H. A. L. Kiers, J. M. F. ten Berge, and R. Bro, PARAFAC2 - Part I. A direct fitting algorithm for the PARAFAC2 model, J. Chemom., 13 (1999), pp. 275--294.
40.
J. Kim and H. Park, Fast nonnegative matrix factorization: An active-set-like method and comparisons, SIAM J. Sci. Comput., 33 (2011), pp. 3261--3281.
41.
T. G. Kolda and B. W. Bader, Tensor decompositions and applications, SIAM Rev., 51 (2009), pp. 455--500.
42.
T. G. Kolda and D. Hong, Stochastic gradients for large-scale tensor decomposition, SIAM J. Math. Data Sci., 2 (2020), p. 1066--1095.
43.
I. Lehmann, E. Acar, T. Hasija, M. A. Akhonda, V. D. Calhoun, P. J. Schreier, and T. Adali, Multi-task fMRI data fusion using IVA and PARAFAC$2$, in Proceedings of ICASSP'22, 2022, pp. 1466--1470.
44.
K. Madsen, N. Churchill, and M. Mørup, Quantifying functional connectivity in multi-subject fMRI data using component models, Human Brain Mapping, 38 (2017), pp. 882--899.
45.
T. Maehara, K. Hayashi, and K. Kawarabayashi, Expected tensor decomposition with stochastic gradient descent, in Proceedings of AAAI'16, 2016, p. 1919--1925.
46.
M. Mardani, G. Mateos, and G. B. Giannakis, Subspace learning and imputation for streaming big data matrices and tensors, IEEE Trans. Signal Process., 63 (2015), pp. 2663--2677.
47.
D. Mitchell, N. Ye, and H. De Sterck, Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms, Numer. Linear Algebra Appl., 27 (2020), e2297.
48.
M. Mørup, L. K. Hansen, C. S. Herrmann, J. Parnas, and S. M. Arnfred, Parallel factor analysis as an exploratory tool for wavelet transformed event-related EEG, NeuroImage, 29 (2006), pp. 938--947.
49.
D. Nion and N. D. Sidiropoulos, Adaptive algorithms to track the PARAFAC decomposition of a third-order tensor, IEEE Trans. Signal Process., 57 (2009), pp. 2299--2310.
50.
E. E. Papalexakis, C. Faloutsos, and N. D. Sidiropoulos, Tensors for data mining and data fusion: Models, applications, and scalable algorithms, ACM Trans. Intell. Syst. Technol., 8 (2016), 16.
51.
N. Parikh and S. Boyd, Proximal algorithms, Found. Trends Mach. Learn., 1 (2014), pp. 127--239.
52.
Y. Ren, J. Lou, L. Xiong, and J. C. Ho, Robust irregular tensor factorization and completion for temporal health data analysis, in Proceedings of CIKM'20, 2020, pp. 1295--1304.
53.
M. Roald, S. Bhinge, C. Jia, V. Calhoun, T. Adali, and E. Acar, Tracing network evolution using the PARAFAC2 model, in Proceedings of ICASSP'20, 2020, pp. 1100--1104.
54.
M. Roald, C. Schenker, J. E. Cohen, and E. Acar, PARAFAC2 AO-ADMM: Constraints in all modes, in Proceedings of EUSIPCO'21, EURASIP, 2021.
55.
C. Schenker, J. E. Cohen, and E. Acar, A flexible optimization framework for regularized matrix-tensor factorizations with linear couplings, IEEE J. Sel. Top. Signal Process., 15 (2021), pp. 506--521.
56.
C. Schenker, J. E. Cohen, and E. Acar, An optimization framework for regularized linearly coupled matrix-tensor factorization, in Proceedings of EUSIPCO'20, EURASIP, 2021, pp. 985--989.
57.
A. M. Shun Ang, J. E. Cohen, L. T. Khanh Hien, and N. Gillis, Extrapolated alternating algorithms for approximate canonical polyadic decomposition, in Proceedings of ICASSP'20, 2020, pp. 3147--3151.
58.
Q. F. Stout, Unimodal regression via prefix isotonic regression, Comput. Statist. Data Anal., 53 (2008), pp. 289--297.
59.
J. M. ten Berge and H. A. Kiers, Some uniqueness results for PARAFAC$2$, Psychometrika, 61 (1996), pp. 123--132.
60.
M. H. Van Benthem and M. R. Keenan, Fast algorithm for the solution of large-scale non-negativity-constrained least squares problems, J. Chemom., 18 (2004), pp. 441--450.
61.
M. H. Van Benthem, T. J. Keller, G. D. Gillispie, and S. A. DeJong, Getting to the core of PARAFAC2, a nonnegative approach, Chemom. Intel. Lab. Syst., 206 (2020), 104127.
62.
M. Vandecappelle, N. Vervliet, and L. D. Lathauwer, Nonlinear least squares updating of the canonical polyadic decomposition, in Proceedings of EUSIPCO'17, EURASIP, 2017, pp. 663--667.
63.
Y.-X. Wang and Y.-J. Zhang, Nonnegative matrix factorization: A comprehensive review, IEEE Trans. Knowl. Data Eng., 25 (2012), pp. 1336--1353.
64.
A. H. Williams, T. H. Kim, F. Wang, S. Vyas, S. I. Ryu, K. V. Shenoy, M. Schnitzer, T. G. Kolda, and S. Ganguli, Unsupervised discovery of demixed, low-dimensional neural dynamics across multiple timescales through tensor component analysis, Neuron, 98 (2018), pp. 1099--1115.
65.
K. Yin, A. Afshar, J. C. Ho, W. K. Cheung, C. Zhang, and J. Sun, LogPar: Logistic PARAFAC2 factorization for temporal binary data with missing values, in Proceedings of KDD'20, 2020, pp. 1625--1635.
66.
H. Yu, D. Augustijn, and R. Bro, Accelerating PARAFAC2 algorithms for non-negative complex tensor decomposition, Chemom. Intel. Lab. Syst., 214 (2021), 104312.
67.
B. J. Zijlstra and H. A. Kiers, Degenerate solutions obtained from several variants of factor analysis, J. Chemom., 16 (2002), pp. 596--605.
Information & Authors
Information
Published In
SIAM Journal on Mathematics of Data Science
Pages: 1191 - 1222
DOI: 10.1137/21M1450033
ISSN (online): 2577-0187
Copyright
© 2022, Society for Industrial and Applied Mathematics.
History
Submitted: 1 October 2021
Accepted: 31 May 2022
Published online: 30 August 2022
Keywords
MSC codes
Authors
Funding Information
Research Council of Norway : 300489
Funding Information
Agence Nationale de la Recherche https://doi.org/10.13039/501100001665 : LoRAiA ANR-20-CE23-0010
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Unsupervised EHR ‐based phenotyping via matrix and tensor decompositionsWIREs Data Mining and Knowledge Discovery, Vol. 177 | 5 March 2023
- MatCoupLy: Learning coupled matrix factorizations with PythonSoftwareX, Vol. 21 | 1 Feb 2023