Multivariate Analysis, Dimension Reduction, and Latent Structure
Multivariate distribution theory
Covariance and correlation
For a random vector , the covariance matrix summarizes second-order dependence. Estimation and interpretation become difficult when is large relative to .
Principal component analysis
PCA finds orthogonal directions maximizing variance. It is equivalent to low-rank approximation of the data matrix and can be derived from the eigendecomposition of the sample covariance matrix.
PCA is descriptive
PCA reveals dominant variation but does not by itself establish causation, predictive relevance, or scientific interpretability.
Factor analysis and latent variable models
Latent structure
Factor models explain correlations through a smaller number of unobserved latent variables. If , then contains loadings, latent factors, and idiosyncratic noise.
What does PCA primarily optimize?
PCA selects directions that capture the maximum variance in the data.
Correct answer: Explained variance
What is one difference between PCA and factor analysis?
The two methods can produce similar outputs but have different interpretations and assumptions.
Correct answer: PCA is a deterministic linear dimension-reduction method, while factor analysis posits a probabilistic latent variable model.
Clustering, manifold learning, and modern latent representations
Beyond linear subspaces
Clustering methods such as Gaussian mixture models, spectral clustering, and hierarchical methods infer structure without labels. Manifold learning methods like t-SNE, UMAP, and diffusion maps seek nonlinear low-dimensional embeddings, often for visualization and exploratory analysis.
Comparative roles
Which method is most often used for nonlinear visualization of high-dimensional data?
t-SNE is widely used for visualizing local neighborhood structure in high-dimensional data.
Correct answer: t-SNE
Why should cluster labels be interpreted cautiously?
Unsupervised learning can reveal patterns that are not unique or scientifically meaningful.
Correct answer: Because cluster structure may be unstable, method-dependent, and sometimes driven by noise or scaling choices rather than true latent groups.