AttentionForge implements scaled dot-product attention, multi-head self-attention, and a pre-norm GPT block twice: first as a readable NumPy reference, then as an explicit PyTorch implementation. The CPU suite checks both against independent PyTorch library implementations. Forward-only Triton kernels then apply tiled, online softmax without materializing the full score matrix; v2 also skips K/V tiles that are wholly in the causal future.
The goal is not to beat FlashAttention. It is to understand the math, prove the reference, and profile a learning kernel honestly against mature library code.
| Gate | Status |
|---|---|
| Phase 1 reference and tests | Implemented; local suite passes |
| v1 and v2 correctness on a Tesla T4 | Passed at every timed shape |
| v2 launch study and T4 latency sweep | Measured; hash-backed artifact committed |
| v2 Nsight Compute profile | Captured and interpreted |
| v2 grouped-query attention (GQA/MQA) | Implemented; contract tests pass on CPU. Numerics unverified — no GPU run yet |
| Mayank explanation gate | Not passed |
The latest measured run is 20260822T211657Z on a Tesla T4. Raw tuning samples,
latency samples, environment, backend probes, roofline inputs, Nsight CSV, and
the SHA-256 manifest live in
bench/artifacts/20260822T211657Z. The
manifest hash is 47ab655b224f13fd7e245ba7e26abaaa8586f5a120f86d509886791dd492975c.
The immutable Phase 2 v1 artifact remains at
bench/artifacts/20260822T203543Z.
A later audit found that the original Phase 3 roofline mixed dense v2 FLOPs
with causal tile-visit bytes. The measured latencies and Nsight counters remain
valid, but v2 roofline claims must use the separate
tile-aware correction.
The same audit repeated launch selection with two three-round shuffled studies;
both selected 32x32x4. Full records and their manifest are in the
supplemental T4 hardening bundle.
Configuration: causal fp16 forward attention, B=1, H=8, D=64, 10 warmups,
30 retained timings, seed 1729. All values are median milliseconds; lower is
better. The historical Phase 3 sweep used the fixed 64x64x8 launch selected
by its one sequential tuning pass. The later randomized studies do not rewrite
these rows; they show that the original launch-selection result is not stable.
| Sequence | v0 materialized | v1 Triton | v2 causal-skip | PyTorch efficient |
|---|---|---|---|---|
| 128 | 0.3014 | 0.4260 | 0.2350 | 0.1312 |
| 256 | 0.4484 | 0.7398 | 0.4529 | 0.1593 |
| 512 | 0.6369 | 2.5815 | 0.6437 | 0.1308 |
| 1024 | 1.6229 | 8.5505 | 1.0932 | 0.1872 |
| 2048 | 5.8686 | 34.4750 | 3.7600 | 0.4179 |
v2 is 1.63x to 9.17x faster than v1 and beats materialized v0 at S=128,
S=1024, and S=2048; at S=256 and S=512 it is approximately tied but
slightly slower. PyTorch's separately labeled efficient backend remains 1.79x
to 9.00x faster than v2. The forced Flash backend was unsupported on T4/sm_75,
so there is no FlashAttention latency or gap to report for this artifact.
| Version | Implementation | Purpose | Current evidence |
|---|---|---|---|
| reference | numpy_reference.py |
Stable SDPA, MHA, and GPT block written for readability | CPU tests |
| v0 | explicit PyTorch, materialized scores | Honest baseline that writes the S x S matrix |
Measured on T4 |
| v1 | triton_attention.py |
Tiled online softmax that visits every K/V tile | Correctness-gated and preserved as a baseline |
| v2 | triton_attention.py |
v1 recurrence plus a causal K/V traversal bound | Correctness-gated, tuned, and profiled on T4 |
| library | forced PyTorch SDPA backend | Compare without silent backend fallback | Efficient measured; Flash unsupported |
The Triton study is deliberately narrow: fp16, (B, H, S, D) layout, power-of-two
head widths from 16 through 128, causal or non-causal forward attention, no
dropout, no arbitrary mask, and no backward pass.
v2 additionally accepts grouped-query attention: K and V may carry fewer
heads than Q, with each K/V head shared by num_heads // num_kv_heads query
heads, covering MHA (equal counts), GQA, and MQA (one K/V head). This is the
layout every current production LLM uses, so a kernel restricted to equal head
counts cannot serve one. v1 keeps the original equal-shape contract and is
unchanged.
The head mapping is compile-time (num_kv_heads is constexpr), so MHA reduces
to head_index // 1 and is indexed exactly as before. Correctness is defined
against dense attention with K/V repeated to full head count — an equality that
holds by construction and needs no library feature flag. Non-divisible head
counts and num_kv_heads > num_heads are rejected rather than silently
truncated.
Not yet measured. The GQA path has run only its CPU contract tests. No numerical or latency claim exists for it until a T4 run gates it, and the headline table above contains no GQA rows.
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
python -m pytestThe tests pin seeds and prove:
- explicit SDPA matches
torch.nn.functional.scaled_dot_product_attentionwithin the stated1e-5tolerance; - separate Q/K/V projections match
torch.nn.MultiheadAttentionafter weights are packed explicitly as[W_q; W_k; W_v]; - the NumPy and PyTorch MHA/GPT implementations agree;
- causal attention cannot leak future keys or values;
- attention rows are probabilities and shapes hold across varied batches, heads, sequence lengths, head dimensions, and value dimensions.
Open colab/AttentionForge_T4.ipynb, attach a T4,
and run it top to bottom to reproduce the committed workflow:
- asserts the hardware and captures the environment;
- runs CPU tests;
- correctness-gates six fixed v2 launch candidates at
S=1024, then times them across three deterministically shuffled rounds and records the aggregate median winner; - checks v1 and the fixed v2 launch against forced PyTorch math SDPA at every shape before timing;
- benchmarks materialized PyTorch, v1, v2, and every explicitly supported PyTorch fused backend;
- creates separate v1/v2 roofline byte models and captures v2 Nsight Compute CSV;
- hashes the complete artifact directory.
PyTorch's SDPBackend.FLASH_ATTENTION is forced, never merely requested. If it is
unsupported, the benchmark records the failure reason and emits no Flash row.
This matters because the official FlashAttention-2 CUDA implementation targets
Ampere, Ada, and Hopper; its documentation points Turing/T4 users to a separate
implementation. PyTorch's efficient SDPA backend may run on T4, but it is labeled
torch_sdpa_efficient, never “FlashAttention.” An exact Flash row may require a
separately labeled Ampere-or-newer run; it must not be mixed into the T4 table.
docs/how-it-works.md explains the scaling, exact causal
mask, Q/K/V packing, online-softmax recurrence, v2 causal traversal, roofline
assumptions, profiling questions, and the measured library comparison.
PROJECT_SPEC.md
defines the evidence gates.
The original resume template requires a real FlashAttention comparison. This T4
run cannot fill that field because the forced Flash backend was unsupported.
resume_eligible also stays no until Mayank passes the explanation gate.
Do not use this template yet:
Implemented scaled dot-product attention, multi-head attention, and a GPT block from scratch, verified against
torch.nn.MultiheadAttention(max error [x]); wrote a fused Triton attention kernel with streaming softmax and profiled it on a Tesla T4 — [k]x over naive PyTorch, within [m]x of FlashAttention, with a roofline showing the [memory/compute]-bound regime and achieved DRAM bandwidth.