The Physics of Cache Misses: Memory Latency and Cosmic Distances

A Human Metaphor for Silicon Speed

Humans are notoriously poor at comprehending micro-durations. A nanosecond ($10^{-9}$ seconds) feels identical to a millisecond ($10^{-3}$ seconds). But to a processor executing four cycles per nanosecond, they are different planetary epochs.

Let us scale one CPU cycle (0.3 nanoseconds) to **one human second**:

| Hardware Event | Silicon Time | Human Scaled Time | Metaphorical Distance | | :--- | :--- | :--- | :--- | | **L1 Cache Reference** | 0.5 ns | ~1.5 seconds | Grabbing a pen from your desk | | **Branch Mispredict** | 5 ns | ~15 seconds | Looking around the room | | **L2 Cache Reference** | 7 ns | ~21 seconds | Grabbing a book from the shelf | | **L3 Cache Reference** | 20 ns | ~1 minute | Walking to the lab door | | **Main Memory (DRAM)** | 60–100 ns | ~3 to 5 minutes | Walking down the street for coffee | | **NVMe SSD Read** | 25,000 ns | ~1.5 days | Taking a flight to another continent | | **Network roundtrip (SF to NY)**| 40,000,000 ns| ~4.5 years | Traveling to Jupiter and back |

Why Data Locality Dominates Algorithmic Big-O

In introductory algorithms, we are taught that binary search trees provide $O(\log N)$ operations and arrays provide $O(N)$ insertions. But in practical benchmarks on modern hardware, traversing a pointer-chasing tree can be twenty times slower than scanning a contiguous vector.

Why? Because the tree constantly forces the CPU to wait four minutes for DRAM, while the sequential array allows hardware prefetchers to feed the L1 cache like a conveyor belt.