Dimensionality Reduction: PCA, t-SNE and UMAP, and What Each One Throws Away

Too Many Columns

A grayscale image of 28 by 28 pixels is a point in a space with 784 dimensions. A single cell measured with RNA sequencing is a point in a space with around 20,000, one for each gene. Nobody can look at a scatter plot in 20,000 dimensions, and many algorithms struggle there too.

Richard Bellman called this the curse of dimensionality. As dimensions grow, the volume of the space grows so fast that data becomes sparse. The distance between a point and its nearest neighbor ends up not much smaller than the distance to its farthest one, which undermines anything that depends on «close» meaning something, such as nearest-neighbor methods and clustering.

The saving grace is that real data rarely fills all those dimensions. Pixels in a handwritten digit are strongly correlated with their neighbors, and genes are regulated in coordinated programs. The data usually lies near a much lower-dimensional structure, often called a manifold. Dimensionality reduction tries to find that structure and describe each point with far fewer numbers, while keeping the information that matters. Every method has to decide what matters, and that decision is the main difference between them.

Principal Component Analysis

PCA is the oldest of the three methods here. Karl Pearson described it in 1901, and Harold Hotelling developed it further in 1933. It finds new axes for the data, called principal components, ordered by how much variance they capture. The first component is the direction along which the data spreads out the most. The second is the direction of greatest remaining spread, perpendicular to the first, and so on.

Computationally, these directions are the eigenvectors of the data’s covariance matrix, or equivalently come from the singular value decomposition of the centered data. Keeping only the first few components gives a projection that preserves as much variance as any linear projection to that many dimensions can.

PCA has properties that make it a solid first step. It is deterministic, so the same data always gives the same result. It is fast. The components are linear combinations of the original features, which means they can be inspected: a component might load heavily on a group of genes known to be involved in the cell cycle. And the explained variance of each component tells you how much information you are keeping. A plot of cumulative explained variance is the standard way to decide how many components to keep.

A well-known application is eigenfaces, from Turk and Pentland in 1991: PCA on a set of face images produces components that look like ghostly faces, and any face can be approximately rebuilt as a weighted sum of a few dozen of them. PCA is also used as preprocessing, to compress data before running a slower method on it, and to remove noise by discarding low-variance components.

Its limitation is that it is linear. If the data lies on a curved surface, like a sheet of paper rolled into a spiral, PCA projects it flat and points from different layers of the roll land on top of each other.

Nonlinear Methods Before t-SNE

In 2000, the journal Science published two papers in the same issue proposing ways to «unroll» curved manifolds. Isomap, from Tenenbaum, de Silva and Langford, estimates distances along the manifold by building a neighborhood graph and computing shortest paths through it, then embeds those geodesic distances in low dimensions. Locally Linear Embedding, from Roweis and Saul, describes each point as a weighted combination of its neighbors and finds a low-dimensional layout that preserves those weights. Both worked well on clean synthetic examples like the «Swiss roll,» but they were sensitive to noise and to the choice of neighborhood size on real data.

t-SNE

t-SNE, from Laurens van der Maaten and Geoffrey Hinton in 2008, focused on a narrower goal: making two-dimensional pictures in which points that are neighbors in the original space stay neighbors in the picture.

It converts distances in high dimensions into probabilities that one point would pick another as a neighbor, using a Gaussian centered on each point. It then places points in two dimensions and defines similar probabilities there, but with a Student t-distribution, which has heavier tails. Finally, it moves the points around with gradient descent to make the two sets of probabilities match, measured by KL divergence. The heavy-tailed distribution in the low-dimensional space is the key trick. It lets moderately distant points be placed far apart, which solves the «crowding problem» of the earlier SNE method and produces the well-separated clusters t-SNE plots are known for.

The main parameter is perplexity, which roughly sets how many neighbors each point considers. Wattenberg, Viégas and Johnson’s 2016 article «How to Use t-SNE Effectively» in the journal Distill showed how much the output can change with it, and listed several traps: cluster sizes in a t-SNE plot mean nothing, distances between clusters may mean nothing, and random noise can produce apparent clusters at low perplexity.

UMAP

UMAP, from McInnes, Healy and Melville in 2018, has a theoretical basis in topology (fuzzy simplicial sets), but in practice it works much like t-SNE. It builds a weighted nearest-neighbor graph in high dimensions and optimizes a low-dimensional layout whose graph matches it, using a cross-entropy objective.

It became popular quickly for practical reasons. It is much faster than the original t-SNE and scales to millions of points. It can embed into more than two dimensions, so it can serve as general preprocessing and not only visualization. It can place new points into an existing embedding. And its plots were thought to preserve more global structure, meaning the relative positions of clusters. In 2019, Becht and colleagues published an evaluation in Nature Biotechnology arguing for UMAP in single-cell analysis, and it became the default visualization in tools like Scanpy and Seurat.

Later work complicated the global structure claim. Kobak and Linderman showed in 2021 that a large part of the difference came from initialization: UMAP started from a spectral embedding by default, while t-SNE was usually started at random. When both are given the same informative initialization, the gap in global structure mostly disappears.

The Distortion Problem

In 2023, Tara Chari and Lior Pachter published «The specious art of single-cell genomics» in PLOS Computational Biology. They measured how well two-dimensional t-SNE and UMAP embeddings of real single-cell data preserved local neighborhoods, global relationships between cell types, and pairwise distances. Their conclusion was that the distortion is severe on all three counts. To show how little the local-preservation criterion constrains the picture, they also produced an embedding of the same data forced into the shape of an elephant, which preserved neighborhoods comparably to the standard methods.

This does not make t-SNE or UMAP useless. It means a 2D plot of 20,000-dimensional data is a lossy summary, and conclusions like «these two cell types are close» or «this cluster is twice as spread out» should be checked in the original space or in a PCA space, not read off the picture.

Choosing Between Them

For preprocessing before another model, PCA is usually the right choice: it is fast, reproducible and interpretable, and it keeps a meaningful notion of distance. For compressing data where the structure is strongly nonlinear, autoencoders (neural networks trained to reconstruct their input through a narrow bottleneck layer) are the learned alternative. For visualizing local cluster structure in large datasets, t-SNE and UMAP are both reasonable, ideally run with a few different parameter settings and a PCA initialization. A common workflow in single-cell analysis combines them: reduce to around 50 principal components first, then run UMAP or t-SNE on those.

Reflection

Each method keeps something different: PCA keeps variance, Isomap keeps distances along the manifold, t-SNE and UMAP keep neighborhoods. None of them can keep everything, because squeezing thousands of dimensions into two always loses information. A clean, well-separated scatter plot is easy to read as a faithful picture of the data, and it rarely is. It is safer to know what the method was designed to keep and to trust the picture only on that.

By: Max Johnson B.