Multivariate Analysis and Dimension Reduction
Statistics in multiple dimensions
Means, covariances, and geometry
For a random vector , the mean vector and covariance matrix summarize location and dependence. The covariance matrix is symmetric and positive semidefinite.
Why is the covariance matrix always positive semidefinite?
For any vector , equals the variance of a linear combination and is nonnegative.
Correct answer: Because
What is the goal of principal component analysis?
PCA rotates the data to capture most variability in a few principal components.
Correct answer: To find orthogonal directions that maximize variance and reduce dimensionality.
Eigenstructure and PCA
If , the eigenvectors give principal directions and the eigenvalues give the variance explained in each direction. Truncating to the largest eigenvalues yields a low-dimensional approximation.
In PCA, the first principal component corresponds to the direction with
The first principal component maximizes the variance of projected data under a unit-norm constraint.
Correct answer: Maximum variance
PCA vs factor analysis
PCA
- Descriptive dimension reduction
- Components are linear combinations of observed variables
- Focus on variance capture
Factor analysis
- Latent-variable model
- Assumes shared factors explain correlations
- Focus on modeling covariance structure
What is a common challenge in high-dimensional data analysis?
This leads to singular covariance estimates, overfitting, and the need for regularization or sparsity.
Correct answer: The number of variables may be comparable to or exceed the sample size, making estimation unstable.