Skip to content

About

Attention from scratch, verified before it is optimized: NumPy and PyTorch references, a Triton online-softmax kernel with causal tile skipping and grouped-query attention, benchmarked honestly on a Tesla T4

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

AttentionForge — attention from scratch, verified before it is optimized

AttentionForge implements scaled dot-product attention, multi-head self-attention, and a pre-norm GPT block twice: first as a readable NumPy reference, then as an explicit PyTorch implementation. The CPU suite checks both against independent PyTorch library implementations. Forward-only Triton kernels then apply tiled, online softmax without materializing the full score matrix; v2 also skips K/V tiles that are wholly in the causal future.

The goal is not to beat FlashAttention. It is to understand the math, prove the reference, and profile a learning kernel honestly against mature library code.

Status: resume_eligible: no

Gate Status
Phase 1 reference and tests Implemented; local suite passes
v1 and v2 correctness on a Tesla T4 Passed at every timed shape
v2 launch study and T4 latency sweep Measured; hash-backed artifact committed
v2 Nsight Compute profile Captured and interpreted
v2 grouped-query attention (GQA/MQA) Implemented; contract tests pass on CPU. Numerics unverified — no GPU run yet
Mayank explanation gate Not passed

The latest measured run is 20260822T211657Z on a Tesla T4. Raw tuning samples, latency samples, environment, backend probes, roofline inputs, Nsight CSV, and the SHA-256 manifest live in bench/artifacts/20260822T211657Z. The manifest hash is 47ab655b224f13fd7e245ba7e26abaaa8586f5a120f86d509886791dd492975c. The immutable Phase 2 v1 artifact remains at bench/artifacts/20260822T203543Z.

A later audit found that the original Phase 3 roofline mixed dense v2 FLOPs with causal tile-visit bytes. The measured latencies and Nsight counters remain valid, but v2 roofline claims must use the separate tile-aware correction. The same audit repeated launch selection with two three-round shuffled studies; both selected 32x32x4. Full records and their manifest are in the supplemental T4 hardening bundle.

Headline T4 results

Configuration: causal fp16 forward attention, B=1, H=8, D=64, 10 warmups, 30 retained timings, seed 1729. All values are median milliseconds; lower is better. The historical Phase 3 sweep used the fixed 64x64x8 launch selected by its one sequential tuning pass. The later randomized studies do not rewrite these rows; they show that the original launch-selection result is not stable.

Sequence v0 materialized v1 Triton v2 causal-skip PyTorch efficient
128 0.3014 0.4260 0.2350 0.1312
256 0.4484 0.7398 0.4529 0.1593
512 0.6369 2.5815 0.6437 0.1308
1024 1.6229 8.5505 1.0932 0.1872
2048 5.8686 34.4750 3.7600 0.4179

v2 is 1.63x to 9.17x faster than v1 and beats materialized v0 at S=128, S=1024, and S=2048; at S=256 and S=512 it is approximately tied but slightly slower. PyTorch's separately labeled efficient backend remains 1.79x to 9.00x faster than v2. The forced Flash backend was unsupported on T4/sm_75, so there is no FlashAttention latency or gap to report for this artifact.

What is implemented

Version Implementation Purpose Current evidence
reference numpy_reference.py Stable SDPA, MHA, and GPT block written for readability CPU tests
v0 explicit PyTorch, materialized scores Honest baseline that writes the S x S matrix Measured on T4
v1 triton_attention.py Tiled online softmax that visits every K/V tile Correctness-gated and preserved as a baseline
v2 triton_attention.py v1 recurrence plus a causal K/V traversal bound Correctness-gated, tuned, and profiled on T4
library forced PyTorch SDPA backend Compare without silent backend fallback Efficient measured; Flash unsupported

The Triton study is deliberately narrow: fp16, (B, H, S, D) layout, power-of-two head widths from 16 through 128, causal or non-causal forward attention, no dropout, no arbitrary mask, and no backward pass.

v2 additionally accepts grouped-query attention: K and V may carry fewer heads than Q, with each K/V head shared by num_heads // num_kv_heads query heads, covering MHA (equal counts), GQA, and MQA (one K/V head). This is the layout every current production LLM uses, so a kernel restricted to equal head counts cannot serve one. v1 keeps the original equal-shape contract and is unchanged.

The head mapping is compile-time (num_kv_heads is constexpr), so MHA reduces to head_index // 1 and is indexed exactly as before. Correctness is defined against dense attention with K/V repeated to full head count — an equality that holds by construction and needs no library feature flag. Non-divisible head counts and num_kv_heads > num_heads are rejected rather than silently truncated.

Not yet measured. The GQA path has run only its CPU contract tests. No numerical or latency claim exists for it until a T4 run gates it, and the headline table above contains no GQA rows.

CPU quickstart

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
python -m pytest

The tests pin seeds and prove:

  • explicit SDPA matches torch.nn.functional.scaled_dot_product_attention within the stated 1e-5 tolerance;
  • separate Q/K/V projections match torch.nn.MultiheadAttention after weights are packed explicitly as [W_q; W_k; W_v];
  • the NumPy and PyTorch MHA/GPT implementations agree;
  • causal attention cannot leak future keys or values;
  • attention rows are probabilities and shapes hold across varied batches, heads, sequence lengths, head dimensions, and value dimensions.

Human-started GPU workflow

Open colab/AttentionForge_T4.ipynb, attach a T4, and run it top to bottom to reproduce the committed workflow:

  1. asserts the hardware and captures the environment;
  2. runs CPU tests;
  3. correctness-gates six fixed v2 launch candidates at S=1024, then times them across three deterministically shuffled rounds and records the aggregate median winner;
  4. checks v1 and the fixed v2 launch against forced PyTorch math SDPA at every shape before timing;
  5. benchmarks materialized PyTorch, v1, v2, and every explicitly supported PyTorch fused backend;
  6. creates separate v1/v2 roofline byte models and captures v2 Nsight Compute CSV;
  7. hashes the complete artifact directory.

FlashAttention/T4 compatibility boundary

PyTorch's SDPBackend.FLASH_ATTENTION is forced, never merely requested. If it is unsupported, the benchmark records the failure reason and emits no Flash row. This matters because the official FlashAttention-2 CUDA implementation targets Ampere, Ada, and Hopper; its documentation points Turing/T4 users to a separate implementation. PyTorch's efficient SDPA backend may run on T4, but it is labeled torch_sdpa_efficient, never “FlashAttention.” An exact Flash row may require a separately labeled Ampere-or-newer run; it must not be mixed into the T4 table.

Read before claiming it

docs/how-it-works.md explains the scaling, exact causal mask, Q/K/V packing, online-softmax recurrence, v2 causal traversal, roofline assumptions, profiling questions, and the measured library comparison. PROJECT_SPEC.md defines the evidence gates.

The original resume template requires a real FlashAttention comparison. This T4 run cannot fill that field because the forced Flash backend was unsupported. resume_eligible also stays no until Mayank passes the explanation gate. Do not use this template yet:

Implemented scaled dot-product attention, multi-head attention, and a GPT block from scratch, verified against torch.nn.MultiheadAttention (max error [x]); wrote a fused Triton attention kernel with streaming softmax and profiled it on a Tesla T4 — [k]x over naive PyTorch, within [m]x of FlashAttention, with a roofline showing the [memory/compute]-bound regime and achieved DRAM bandwidth.

About

Attention from scratch, verified before it is optimized: NumPy and PyTorch references, a Triton online-softmax kernel with causal tile skipping and grouped-query attention, benchmarked honestly on a Tesla T4

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages