From spec sheet to meaning

Six rules turn a spec sheet into what you will notice. Every number below is computed from the spec table with the rule shown, so it is derived: an upper bound for speeds and a lower bound for times. Real software lands below a speed bound and above a time bound; how far has to be measured per machine and runtime, and we publish measurements only from hardware we own.

Rule 1. Answer speed: tokens/s ≤ memory bandwidth / bytes read per token

decode tokens/s  ≤  bandwidth (GB/s) / [ active weight bytes + KV-cache bytes at the current context ]

Writing one token of a single answer reads every active weight once, plus the attention cache for the context so far. At "4-bit" we count 0.5625 bytes per parameter assumption [S46] (MLX default 4-bit affine: groups of 64 weights with a 16-bit scale and a 16-bit bias, 4 + 32/64 = 4.5 bits). A mixture-of-experts model reads only its active parameters per token. KV cache per token of context, FP16: 8B dense (Llama 3 8B shape): 131 KB (32 layers x 8 KV heads x head size 128 (4,096 / 32) x 2 (K and V) x 2 bytes) spec [S42]; 30B-A3B MoE (Qwen3-30B-A3B): 98 KB (48 layers x 4 KV heads x head size 128 x 2 x 2 bytes) spec [S43] [S44]; 70B dense (Llama 3 70B shape): 328 KB (80 layers x 8 KV heads x head size 128 (8,192 / 64) x 2 x 2 bytes) spec [S42].

Short context

Tokens per second, one user, 4-bit weights; "does not fit" by rule 2.

At 32,768 tokens of context (document-sized prompts)

Tokens per second, one user, 4-bit weights; "does not fit" by rule 2.

For long prompts the cache read can be as large as the weights, so the ceiling roughly halves. Document Q&A lives in the second table.

Rule 2. What fits: largest model ≈ usable memory / bytes per parameter

Usable share: Macs 75% assumption [S47] (macOS caps the GPU's wired memory; about three quarters by default on large-memory Macs (check recommendedMaxWorkingSetSize; raise with sudo sysctl iogpu.wired_limit_mb=N)); DGX Spark 75% assumption (NVIDIA publishes no share; its 'up to 200B parameters' claim for 128 GB implies more); Ryzen AI Max: the GPU-assignable memory AMD publishes spec [S23] [S25]; RTX cards 90% assumption (card memory is not shared with the OS; the rest covers activations and the CUDA context). Then subtract the KV cache for your context. Mixture-of-experts models fit by their total parameters and run (rule 1) by their active ones.

Vendor claims run higher (NVIDIA: up to 200B on a 128 GB Spark spec [S14]; AMD: up to 128B in its 2025 FAQ and "up to 200 billion" on the Ryzen AI Halo in 2026 spec [S23] [S25]). They appear to assume about 4 bits per weight or less and little room for the cache; the vendors do not state their assumptions.

Rule 3. Reading the prompt: time ≥ 2 × active parameters × prompt tokens / peak compute

A prompt is processed all at once, so arithmetic limits it, not bandwidth. Only machines with a published dense FP16/BF16 peak can be put through the rule; the attention term is ignored, so these are floors. Example: a 16,000-token prompt (a few documents). Floors are rounded down.

GeForce cards run FP16 with FP16 accumulation at twice the rate above, and FP8/FP4 paths are faster again where the runtime uses them. Sparse figures need structured sparsity that ordinary weights do not have. Mac prefill has to be measured.

Rule 4. Fine-tuning: memory sets what you can train

Bytes per parameter for the model states, before activations: full fine-tuning with mixed-precision Adam 16 cited [S48]; LoRA on a BF16 base about 2 assumption; a 4-bit base about 0.5625 assumption. Activations come on top, so practical limits are lower.

Rule 5. More than one unit

Tensor-parallel splits (a slice of every layer per unit) raise the decode ceiling with the number of units, minus the time spent exchanging results over the link after every layer. Pipeline splits add capacity, but one answer still passes through every block in turn, so its ceiling stays at one unit's. Links:

• GeForce RTX 5090: PCIe Gen 5; no NVLink spec [S35]
• Mac Studio, M5 Ultra (30-core CPU / 64-core GPU, or 36 / 80): Thunderbolt 5, 80 Gb/s each way symmetric (120 Gb/s one way in Bandwidth Boost); RDMA clustering spec [S1] [S2] [S56]
• GeForce RTX 4090 (previous generation): no NVLink spec [S35]
• Mac Studio, M5 Max 18-core CPU / 40-core GPU: Thunderbolt 5, 80 Gb/s each way symmetric spec [S1] [S56]
• DGX Spark, 128 GB (NVIDIA and OEM GB10 versions): ConnectX-7, 200 Gb/s (25 GB/s), two QSFP ports; up to four units spec [S14] [S15]
• Ryzen AI Max+ PRO 495 systems (AMD Ryzen AI Halo next-gen, Framework Desktop 192 GB): USB4 40 Gb/s; PCIe x4 slot on Framework's board spec [S28]
• Ryzen AI Max+ 395 systems (Framework Desktop, AMD Ryzen AI Halo, others): USB4 40 Gb/s (two ports); 10 GbE on Ryzen AI Halo spec [S25]

Rule 6. Power: read the whole system, not the chip

• GeForce RTX 5090: 575 W card; 1,000 W PSU recommended spec [S33] (card only; add the host PC)
• RTX PRO 6000 Blackwell: 600 W spec [S40] (card only)
• Mac Studio, M5 Ultra (30-core CPU / 64-core GPU, or 36 / 80): 9 W idle / 385 W max for 36/80-core, 512 GB, 16 TB; 480 W enclosure max continuous spec [S1] [S7] (whole computer)
• GeForce RTX 4090 (previous generation): 450 W card; 850 W PSU minimum spec [S34] (card only)
• RTX PRO 4000 Blackwell: 145 W, single slot spec [S39] (card only)
• Mac Studio, M5 Max 18-core CPU / 40-core GPU: 480 W enclosure max continuous; Apple's power page lists only the 32-core configuration spec [S1] [S7] (whole computer)
• MacBook Pro 14/16, M5 Max 40-core GPU: 96 W (14-inch) / 140 W (16-inch) adapter spec [S4] (adapter rating)
• Mac Studio, M5 Max 18-core CPU / 32-core GPU: 7 W idle / 200 W max spec [S7] (whole computer)
• Mac mini, M5 Pro: 155 W max continuous spec [S5] (whole computer)
• DGX Spark, 128 GB (NVIDIA and OEM GB10 versions): GB10 TDP 140 W; 240 W external supply spec [S14] [S15] (whole computer)
• DGX Spark, 64 GB (partners only): as the 128 GB unit spec [S15] (whole computer)
• Ryzen AI Max+ PRO 495 systems (AMD Ryzen AI Halo next-gen, Framework Desktop 192 GB): cTDP 45-120 W; Halo TDP 120 W spec [S24] (chip only; add board, SSD, fans)
• Ryzen AI Max+ 395 systems (Framework Desktop, AMD Ryzen AI Halo, others): cTDP 45-120 W spec [S22] (chip only; add board, SSD, fans)

These figures have different bases, so compare like with like. We do not convert them to energy per token: that needs measured power and measured tokens/s on the same run.

Memory cost per GB derived

Price / memory, for the lowest listed configuration with that memory unless noted. It says what memory costs, nothing about speed.