Interlude show: Memory Wall research

Because a transistor gets cheaper and faster when it shrinks, while a memory access is mostly long wires, and wires don't.

• Energy. Switching costs ½·C·V². Until 2005, every shrink cut both C and V, so energy per switch fell as the cube of the shrink. A DRAM fetch is wordlines, bitlines, a bus across the die and a trace across the board. That wire's capacitance per mm is fixed and its length never shrank.
• Time. DRAM access went from 300 ns to 30 ns while the clock period went from 1,000 ns to 0.2 ns. A miss that cost one cycle in 1980 costs 300–500 now.
• The ratio keeps widening. A 64-bit DRAM read cost ~100× a 32-bit multiply in the mid-1980s and ~500× in 2014. In 2026 it is ~3,000× from DDR5, or 500–700× from HBM.

It points the right way but is incomplete. The energy to move a bit is (energy per mm) × (mm travelled), and both factors stopped improving.

• What 2D/1D gets right: bits fill an area while wires cross a length. So the cost of a random access grows as √(memory size).
• Correction 1: the voltage floor comes from the 60 mV-per-decade subthreshold slope, not from thermal noise. Five decades of on/off ratio plus speed needs ~0.6–0.75 V.
• Correction 2: the DRAM cell is its own wall. Charge sensing and a ~3 V wordline set its latency, and wire physics alone doesn't explain that.
• Detail: ruthenium is the liner and, increasingly, a barrier-free conductor; tantalum nitride is the barrier it replaces.

Capacitance depends on the shape of the conductors, not their size. A shrink keeps the shape, so capacitance per mm stays put, while resistance per mm climbs.

• Why they cancel: capacitance per length ≈ ε × (height ÷ spacing). The wire's facing area shrinks, but so does the gap to its neighbour. Capacitance scales with linear size (a sphere's grows with its radius), so per unit length it depends only on shape.
• The only knob is ε, the insulator. Its k went from 3.9 toward ~2.5, then stuck around 2011, when porous films became too fragile to manufacture.
• Resistance has no such luck: resistance per mm is ρ ÷ (width × height), so it rises as the square of the shrink. Below ~12 nm, copper's resistivity itself climbs 4–6×.
• Short local wires shrink in length too, so they get cheaper. A wire that must cross a die or a board keeps its length, and with it its capacitance.

Capacitance is how much charge a structure must hold to sit at a given voltage (C = Q/V). Sending a 1 means pushing that charge onto a wire, which costs time (≈ R·C) and energy (≈ ½·C·V²), however clever the circuit.

• A transistor's capacitance shrinks with it, which is why arithmetic got thousands of times cheaper. A wire's capacitance grows with its length.
• In DRAM the bit is a charge. A 10–20 fF cell is read by sharing its charge with the bitline. That step is an RC process no transistor shrink speeds up, so it sets the latency floor.

Capacity scaled spectacularly and speed did not. Each time the gap grew about 5×, another cache level appeared.

One more level every time the gap grows ~5×

Rule of thumb from the report: levels ≈ log(gap) ÷ log(5), where the gap is DRAM access time ÷ clock period.

Sixty years in twelve steps

• 1965Wilkes proposes "slave memories", the idea of a cache.
• 1966–68Dennard invents the one-transistor DRAM cell at IBM.
• 1968IBM S/360-85 ships the first commercial cache, 80 ns against ~1 µs core.
• 1970Intel 1103: 1 Kbit of DRAM at ~300 ns. It kills core memory.
• 1974The Dennard scaling paper. Thirty years of free transistor gains begin.
• 1989Intel 486 puts an 8 KB L1 on the chip.
• 1995Wulf & McKee name the problem in "Hitting the Memory Wall".
• 2001Ho, Mai & Horowitz, "The Future of Wires".
• 2004–05Dennard scaling ends. Voltage stops falling, so wire energy stops falling.
• 2013The HBM standard: memory moves onto the package.
• 2022Stacked 3D V-Cache and FlashAttention: hardware stacks, algorithms count bytes.
• 2026HBM4 is in production at all three vendors; DRAM contract prices rise ~90% in one quarter.

Four walls stack on top of each other: the DRAM cell, the on-chip wire, the off-chip link, and the way memory is organized.

The 15–20× gain from DDR3 to HBM3E came almost entirely from geometry, as the distance shrank from centimetres to millimetres. Transistor scaling contributed almost nothing.

Nothing in the physics reverses the trend. Energy per bit falls only where the wire gets shorter, and latency doesn't fall at all.

• Will not change: the 45–50 ns row cycle (it is physics), wire energy per mm at a fixed voltage, and the direction of the ratio.
• Baseline: bandwidth doubles per generation by widening and stacking, energy per bit halves every 4–5 years, and algorithms carry the rest.
• Upside: DRAM-on-logic arrives by ~2031, bringing ≥5× in energy and bandwidth. Downside: HBM keeps absorbing wafers, and memory gets rationed the way compute once was.

Every Dally number is a capacitance or an RC statement in disguise.

• Your latency matched by coincidence: two ~0.73× factors cancelled. You charged the mean distance and he charged the far one, and your wire was 0.75× his speed.
• The velocity is a stable constant. Repeaters make delay linear in length (Bakoglu: T = 2.5·√(R₀C₀·rc)·L). r·c got worse and the repeaters got faster, so ~c/120 has held for 20 years.
• Mini-report: it rebuilds every figure in the article from six constants: c, r, V, the inverter time constant, SRAM density and off-chip link energy. It flags two inferences. His 15 mm is roughly an 88th-percentile distance, and his 230× only reproduces if the sub-array read and the add are counted on both sides.

32-bit words on a 1-µm grid at 1 fJ per word per µm is Dally's own constant, to within 4%. Reads pay the round trip, writes are posted, and a register-like floor fixes the near cells.

• "The memory wall is really a load wall." A load waits for the round trip; a store is posted and lets the core continue.
• Writes hide their real cost elsewhere: allocation (a write first reads the whole line) and invalidation of every other copy. In a coherent mesh, that control traffic is about a third of all messages.
• Why a floor: without one, the cells next to the processor were 16× cheaper than registers and 160× cheaper than SRAM. That biased every comparison toward many tiny nearby accesses.
• Coarser cells (32 bytes per node) save only ~1.5× per useful byte. The √-distance term is untouched.

At the talk's 100 fJ per bit·mm, every energy is a distance, and every distance squared is an area.

• The real die limit is energy, not the reticle. Beyond ~25–28 mm, a die spends more energy reaching its own edge than an HBM access costs: a 48 mm round trip ≈ 4.8 pJ/bit against 5 pJ/bit for HBM.
• Hierarchies come out geometric. Each tier is ~30× the area and ~10× the energy of the one below.
• 3D stacking is the only way past. A 50–100 µm vertical hop takes the ~10 pJ/bit end to end down to ~2 pJ/bit. That is the physics behind his "10× bandwidth" slide.
• Calibration note: the talk's 100 fJ/bit·mm includes routers and pipeline flops; the article's 30 is the bare wire. Don't double-count.

Charge the off-chip crossing as a constant, ~10 pJ per bit, which is worth about 100,000 hops. MNIST needs about half a megabyte of model.

• The tape is 55 MB: 60,000 training images with labels, then 10,000 test images. A tape is single-pass, so anything seen twice must be kept in the scratchpad.
• The balance point is 0.25–0.5 MB of model, where reading parameters costs as much as streaming the data in. The MLP-128 sits right on it.
• Keeping the training set (1.5% of a die) makes extra epochs 44× cheaper than re-streaming. k-NN costs ~0.86 J, about 60× the MLP's entire run.
• Crossing costs by technology: die-to-die 0.3–1 pJ/bit, HBM 5, DDR 10, PCIe 8–15, and SSDs or networks ~100 pJ/bit.

Use a lattice of tiles with one processor each, and make the pitch p the only parameter. Feed the tapes from a ribbon along the bottom edge, the way real chips use their beachfront.

• A processor per word isn't buildable. A minimal 32-bit core is ~1,000 µm², the area of 1,000 of your words. The "spatial computer" papers assume processors are free.
• p = 128 is the physically honest choice; p = 32 is the GPU-native one. Run it SPMD and bulk-synchronous: one kernel per round, with depth = the number of rounds.
• What the A100 can't show: every message between SMs goes through L2 at one flat cost, so distance beyond a tile is invisible. The traffic volume that crosses tiles does carry over, so algorithm rankings agree.
• Replace the time limit with leakage: charge static power for every powered tile (SRAM ~0.5 W per GB). Time then enters the score through physics.

It is a 1,088-core RISC-V inference chip that bought a huge scratchpad (160 MB of SRAM) and a thin memory pipe (137 GB/s).

• Why it lost: ~1,000 ops per DRAM byte suits recommendation and CNN models that fit in SRAM. It is fatal for LLM decode, where a 7B int8 model tops out near 20 tokens/s per chip.
• Its real strength: models up to ~150 MB stay resident and run at batch 1 at full utilization (its on-chip ridge is ~63). The cores are general, so sparse and irregular work runs too.
• In grid terms: Esperanto bought a bigger scratchpad and a cheaper hop; Thor bought a wider ribbon and cheaper MACs. Which wins depends only on whether the working set fits.
• Three answers to the same physics: Esperanto lowered the voltage and kept everything programmable. NVIDIA kept the voltage high and won on precision and software. Apex (Hasan Unlu) freezes the transformer operators in silicon.
• For modelling: it is a two-level tile machine (shires at ~4 mm, minions at ~600 µm) with DRAM ribbons on the left and right edges. Its caches are not coherent.

Decode runs at 1–70 FLOPs per byte, while every chip's ridge sits in the hundreds or thousands. Only prefill and diffusion steps saturate the hardware.

• The ridge = peak FLOPs ÷ DRAM bandwidth. Below it, the arithmetic units wait on memory.
• Correction you got that day: Thor's FP8 dense ridge is ~1,900, not 3,800. The 3,800 figure is FP4 dense or FP8 sparse.
• Saturating Thor takes reuse: ~1,900 tokens through each weight load (prefill, vision encoders, multi-camera batches). Decode can't get there; a 7B FP8 model runs ~39 tokens/s at 273 GB/s.
• Diffusion LMs turn decode into prefill-shaped work. They are faster at batch 1 but cost S× the FLOPs. They save energy only with few steps (~S < 25 for a 256-token block on HBM), and never when the weights already sit in SRAM.

Yes for the SRAM-centric chips. GPUs were caught too, but they could grow their DRAM. "50 GB at 500K tokens" fits an 8B GQA model; a dense 70B needs 160–180 GB.

• Why it broke the design point: weights are read once per step and shared across the batch. KV is per sequence and re-read every token, so at long context it sets both capacity and bandwidth.
• The 2019–21 chips were designed around BERT at 512 tokens (~19 MB of KV). A 500K context is 1,000–10,000× that, per sequence, and it can't be batched away.
• KV catches up with the weights at ~125K tokens for Llama-8B, ~440K for 70B, and ~10M for DeepSeek-V3. That last figure is MLA's real achievement.
• Every architecture since Llama-2 cuts KV: GQA (4–8×), MLA (~15×), local/global layers (5–20×), linear-attention hybrids (10–30×), FP8 or INT4 KV (2–4×).
• For your grid, this is a second tape: a per-sequence sweep of 24–500 KB per context token, sized by the user rather than the model.

Your numbers all check out. The power argument was about fitting a PCIe slot, not the electricity bill. And Mojo isn't a device: it's Modular's language, which Qualcomm now owns.

• On power: Ditzel never argued electricity cost. A PCIe slot offers 75–120 W, so six low-voltage chips per slot beat one hot chip. Your ~10%-of-TCO point is right for GPUs.
• What changed by 2026: grid power now caps data-centre capacity, so perf per watt counts as capacity rather than opex. That only helps parts whose absolute throughput is competitive, and the ET-SoC-1's isn't. At the edge, power always mattered, which is where Ainekko took it.
• Mojo, as reported in the chat: Qualcomm closed its ~$3.9B purchase of Modular on 29 Jul 2026. Mojo 1.0 shipped on 11 Aug, and the compiler went open source (Apache 2.0) a week later. Chris Lattner is now Qualcomm's EVP for AI software.

On a phone, swipe the table sideways to see all four columns.

Mojo takeaways for your project

1. It's the practical alternative to emitting PTX. Generate Mojo from the grid compiler and the same code runs on A100/H100, AMD, Apple and Thor. It buys you vendors, not fidelity to the grid.
2. The open compiler is the real change. An ET-SoC-1 backend is feasible in principle, but it would be a project, not a configuration.
3. Its pitch is the memory wall in software: it moves work between bandwidth-rich and compute-rich parts, the same prefill/decode split as your ridge tables.
4. Treat its parity claims as claims. Check the one number that matters to you, energy per token at fixed accuracy, on your A100.
5. Ownership cuts both ways. A silicon vendor now sets the roadmap, which the Apache license partly offsets.