Questions tagged [dimensionality-reduction]

Techniques for reducing a large number of variables or dimensions spanned by data to a smaller number of dimensions while preserving as much information about the data as possible. Prominent methods include PCA, Factor Analysis, MDS, Independent Component Analysis, Multiple Correspondence Analysis, Isomap, etc. The two main subclasses of techniques: feature extraction and feature selection.

Techniques for reducing a large number of variables or dimensions spanned by data to a smaller number of dimensions while preserving as much information about the data as possible. Prominent methods include PCA, Factor Analysis, MDS, Independent Component Analysis, Multiple Correspondence Analysis, Isomap, etc. The two main subclasses of techniques: feature extraction and feature selection.

1119 questions
639
votes
5 answers

Relationship between SVD and PCA. How to use SVD to perform PCA?

Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix $\mathbf X$. How does it work? What is the…
amoeba
  • 110,174
176
votes
1 answer

How to reverse PCA and reconstruct original variables from several principal components?

Principal component analysis (PCA) can be used for dimensionality reduction. After such dimensionality reduction is performed, how can one approximately reconstruct the original variables/features from a small number of principal components?…
amoeba
  • 110,174
137
votes
4 answers

PCA and proportion of variance explained

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a precise mathematical definition of what "variance…
user9097
  • 3,393
124
votes
2 answers

Why do we need to normalize data before principal component analysis (PCA)?

I'm doing principal component analysis on my dataset and my professor told me that I should normalize the data before doing the analysis. Why? What would happen If I did PCA without normalization? Why do we normalize data in general? Could someone…
jjepsuomi
  • 6,067
110
votes
13 answers

Explain "Curse of dimensionality" to a child

I heard many times about curse of dimensionality, but somehow I'm still unable to grasp the idea, it's all foggy. Can anyone explain this in the most intuitive way, as you would explain it to a child, so that I (and the others confused as I am)…
Kobe-Wan Kenobi
  • 2,887
88
votes
3 answers

What is the difference between ZCA whitening and PCA whitening?

I am confused about ZCA whitening and normal whitening (which is obtained by dividing principal components by the square roots of PCA eigenvalues). As far as I know, $$\mathbf x_\mathrm{ZCAwhite} = \mathbf U \mathbf x_\mathrm{PCAwhite},$$ where…
RockTheStar
  • 13,247
59
votes
4 answers

Would PCA work for boolean (binary) data types?

I want to reduce the dimensionality of higher order systems and capture most of the covariance on a preferably 2 dimensional or 1 dimensional field. I understand this can be done via principal component analysis, and I have used PCA in many…
Alvin Nunez
  • 777
  • 1
  • 7
  • 8
46
votes
2 answers

What is "reduced-rank regression" all about?

I have been reading The Elements of Statistical Learning and I could not understand what Section 3.7 "Multiple outcome shrinkage and selection" is all about. It talks about RRR (reduced-rank regression), and I can only understand that the premise is…
cgo
  • 9,587
45
votes
3 answers

Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using only 2 principal components does not yield distinct…
willk
  • 653
45
votes
1 answer

What is the intuitive reason behind doing rotations in Factor Analysis/PCA & how to select appropriate rotation?

My Questions What is the intuitive reason behind doing rotations of factors in factor analysis (or components in PCA)? My understanding is, if variables are almost equally loaded in the top components (or factors) then obviously it is difficult to…
GeorgeOfTheRF
  • 5,773
41
votes
2 answers

Intuitive explanation of how UMAP works, compared to t-SNE

I have a PhD in molecular biology. My studies recently started to involve high dimensional data analysis. I got the idea of how t-SNE works (thanks to a StatQuest video on YouTube ) but can't seem to wrap my mind around UMAP (I listened to the UMAP…
Atakan
  • 801
40
votes
2 answers

How would PCA help with a k-means clustering analysis?

Background : I want to classify the residential areas of a city into groups based on their social-economic characteristics, including housing unit density, population density, green space area, housing price, number of schools / health centers / day…
enaJ
  • 625
37
votes
2 answers

When is t-SNE misleading?

Quoting from one of the authors: t-Distributed Stochastic Neighbor Embedding (t-SNE) is a ( prize-winning ) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. So it sounds…
Frames Catherine White
  • 3,524
34
votes
4 answers

Is PCA always recommended?

I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no. If we perform PCA then we calculate linear combinations of the features to build…
Brandon
  • 754
  • 1
  • 5
  • 15
34
votes
1 answer

Dimensionality reduction (SVD or PCA) on a large, sparse matrix

/edit: Further follow up now you can use irlba::prcomp_irlba /edit: following up on my own post. irlba now has "center" and "scale" arguments, which let you use it to calculate principle components, e.g: pc <- M %*% irlba(M, nv=5, nu=0,…
Zach
  • 25,116
1
2 3
…
74 75