Principal Component Analysis
Reduce dimensionality by projecting data onto principal components.
Overview
Principal Component Analysis (PCA) is a dimensionality reduction technique that transforms a dataset with possibly correlated features into a set of linearly uncorrelated variables called principal components. These components are ordered so that the first few retain most of the variation present in the original data.
Learning Objectives
Understand eigenvalue decomposition for PCA
Apply PCA for dimensionality reduction
Interpret explained variance ratios
Understand eigenvalue decomposition for PCA
Apply PCA for dimensionality reduction
Interpret explained variance ratios
Visualize high-dimensional data in 2D/3D using PCA
Prerequisites
Why PCA Exists
The curse of dimensionality and why we need PCA
Curse of Dimensionality
As dimensions increase, data becomes sparse. Distances become meaningless. Models need exponentially more data.
Feature Correlation
Many features are redundant or highly correlated. PCA removes this redundancy by creating uncorrelated components.
Visualization
Human vision works in 2D/3D. PCA reduces high-dimensional data so we can see patterns, clusters, and outliers.
Visual Intuition
See PCA in action
Principal Component Analysis
See how PCA finds directions of maximum variance
What to observe
The eigenvectors (red and blue arrows) point in the directions of maximum variance. PC1 captures the most spread, PC2 captures the next. Watch how data points project onto these axes when you click Play. The percentages show how much variance each component explains.Covariance Matrix
Understanding how features vary together
Data Distribution
How features co-vary determines PCA directions
Covariance Matrix
cov(X,Y) = (1/(n-1)) Σ(xi - x̄)(yi - ȳ)
PCA Eigendecomposition
Σ = QΛQᵀ where Q = eigenvectors, Λ = eigenvalues
Mathematical Explanation
The PCA Algorithm — Step by Step
- Standardize the data (subtract mean, divide by standard deviation)
- Compute the covariance matrix Σ of the standardized data
- Compute eigenvectors and eigenvalues of Σ
- Sort eigenvectors by eigenvalues in descending order
- Select the top k eigenvectors as principal components
- Project the data onto the selected components
Projection Formula
Z = X · W where Z = projected data, X = standardized data, W = top k eigenvectors
Explained Variance Ratio
variance_ratio = λᵢ / Σ(λⱼ) for j = 1 to d
where λ are the eigenvalues and d is the number of dimensions
Worked Example
Walk through a concrete example step by step
Example: Reducing 2D to 1D
Consider three data points in 2D: (1,2), (2,3), (3,4). These points lie roughly along a diagonal.
- Standardize: Mean is (2,3), variance is (1,1) → standardized: (-1,-1), (0,0), (1,1)
- Covariance matrix: [[1, 1], [1, 1]] — features are perfectly correlated
- Eigenvalues: λ₁ = 2, λ₂ = 0
- Eigenvectors: v₁ = [0.707, 0.707]ᵀ (45° direction)
- Projection: Project onto v₁ → points become: -1.414, 0, 1.414
- 100% of variance is captured by the first principal component!
Common Mistakes
Avoid these pitfalls
Forgetting to standardize data before PCA (features on different scales distort results)
Assuming PCA components are interpretable (they are linear combinations, not original features)
Retaining too few components and losing important information
Using PCA on categorical data (PCA assumes continuous, normally distributed features)
Thinking PCA is feature selection (it's feature extraction — components are new features)
Real-World Analogy
Understanding PCA through everyday examples
The Shadow Analogy
Imagine a 3D object. When you shine a light on it from different angles, you get different 2D shadows. PCA finds the best angle to cast the shadow so that the shadow reveals the most information about the object. The direction of the light is the principal component.
The Pizza Box Analogy
If you have a stack of pizza boxes, measuring the height of the stack tells you how many boxes there are (1 dimension captures most info). The length and width of individual boxes are less important. PCA discovers that "height" is the most informative measurement — the first principal component.
Knowledge Graph
See how this concept connects to others
ML Pipeline Flow
Understand where this fits in the ML workflow
Step Details
Click any step to see details
Flowchart
Algorithm workflow
Interactive flowchart ready
Mind Map
Concept connections
Interactive mind map ready
Interactive Playground
Experiment with parameters in real time
PCA Parameter Explorer
Adjust the data and see how PCA responds
Practice Quiz
Test your understanding
Previous Year Questions
Practice with real IITM exam questions
Explain the steps involved in performing PCA on a dataset. How do you determine the number of principal components to retain?
Compare PCA with t-SNE for dimensionality reduction. When would you use one over the other?
A dataset has 100 features. After applying PCA, the first 10 components explain 95% of the variance. What does this imply? How many components would you use?
Quick Revision
Key points to remember
Purpose
Dimensionality reduction that preserves maximum variance
Key Math
Eigendecomposition of the covariance matrix
Output
Principal components — orthogonal directions of maximum variance
Variance Explained
λᵢ / Σ(λ) — tells you how much information each component captures
Assumptions
Linearity, large variance = important, features are continuous
Limitations
Linear only, sensitive to scaling, components are hard to interpret