Sign in Subscribe
Logging · 001

Engineering's view on local frontier AI — what actually runs on hardware you control.

A weekly engineering journal benchmarking open-weights models on consumer silicon. Real harnesses, real failures, no vendor decks.

Recent dispatches
  1. 1 I Recommended the 96 GiB VRAM Carveout. Then a 21 GiB Model Hung on Load. In May I recommended the max carveout and --no-mmap. In June a 21 GiB model hung. Upstream llama.cpp now skips mmap on ROCm and Vulkan iGPUs by default. 7 min read · Aug 27
  2. 2 Speculative Decoding Across My Fleet: Gains From +200% to -80%, and No Single Setting Fits Every Model Speculative decoding swept across the fleet: three draft mechanisms, per-model sweet spots from depth 3 to 5, one model that triples, and one that breaks. 8 min read · Aug 20
  3. 3 Quantized KV Cache Helps Six of My Nine Models and Actively Hurts the Two Newest Ones Nine models on one pinned build, three cache types each. Where quantizing the KV cache pays, where it costs, and what runs at all on 128GB. 9 min read · Aug 13
  4. 4 I Rebuilt the Box, and the Cliff Was Gone. Then I Spent an Evening Proving Which Patch Did It A Fedora upgrade turned into an attribution run. Five builds from one base, and the one patch that removes the dense flash attention collapse at depth. 8 min read · Aug 06
  5. 5 A Reader Fixed Quantized KV Cache on Strix Halo. I Verified It at 262k Context, and His Theory About My Dense Cliff Survived Half Its Test A reader's fork makes quantized KV cache faster than f16 on Strix Halo. Verified on a 128GB box up to the full 262k native context. His cache-spill theory for the dense cliff survived half its test, and the failed half is the interesting part. 6 min read · Jul 30
  6. 6 Turning Off the IOMMU Made My Dense Model 37% Faster. Five More Strix Halo Tips, Measured, and One Crashed the GPU Two guides said turn the IOMMU off. I was the outlier running it on. Measured: +34 to +38% dense prompt processing. Plus five more tips, mostly busted, and one that crashes the GPU. 5 min read · Jul 23
  7. 7 I Measured the Strix Halo Tuning Tips. One Flag Matters More Than All of Them Same box, same build, one flag: +41% for the MoE model, -51% for the dense one. The Strix Halo tuning tips, measured and ranked, including the ones that failed. 8 min read · Jul 16
  8. 8 One llama.cpp Update Made My MoE Model 38% Faster and My Dense Model 45% Slower Same box, same models, same flags, three builds in three weeks. What moved, what broke, and the routine that keeps my numbers honest. 6 min read · Jul 09
  9. 9 The Strix Halo Reference Setup Pack A proven production setup you can copy: BIOS values, kernel params, working systemd units, and real benchmark numbers. 14 min read · Jul 07
  10. 10 Buy or Wait: Reading the Local LLM Hardware Question in a Memory Crunch The instinct is to wait for the next box. In a memory crunch, that's backwards. How to read the buy-or-wait question for Strix Halo, DGX Spark, and Mac Studio. 6 min read · Jul 02
  11. 11 What to Buy for Local LLM Inference: Strix Halo, Mac Studio, DGX Spark, or a GPU Rig A buyer's guide to local LLM hardware, ranked by the one spec that actually decides generation speed: memory bandwidth. Plus where ROCm really stands on Strix Halo. 13 min read · Jun 25
  12. 12 AMD Is Selling "First-Class ROCm" on Strix Halo. I've Run the Same Chip for Six Months. On June 8, AMD opened pre-orders for a $3,999 box built around the exact chip I've run in production since the start of the year, marketed on full ROCm support, with one of its own demos running on the exact model my board can't load under ROCm. 8 min read · Jun 18
  13. 13 A BIOS Update Won't Fix #6182 — I Tried the Newest One The Bosgame M5's ROCm bug is board-specific, not chip-specific — so firmware is the obvious lever. I flashed Bosgame's newest official BIOS hoping to dodge it. It didn't work, and the negative narrows where the fault actually lives. 4 min read · Jun 11
  14. 14 Full Context on a Vulkan-Only Strix Halo: The Decode-Drop Reproduces, but the Sweet Spot Moves kmarble showed ROCm decode collapses 64% at full context on Strix Halo, and ROCm+MTP cures it. My board can't run ROCm. The Vulkan half reproduces the drop — but the MTP sweet spot from last week walks left at depth: by 76k, drafting too deep is slower than no speculation at all. 12 min read · Jun 04
  15. 15 MTP Defaults Are a Trap: What 260 Runs Showed About Speculative Decoding on Qwen3.6 Until May 19, the llama.cpp speculative-decoding default was 16. On Qwen3.6's single MTP head, that default cost up to 75% of generation throughput. Here's where the real sweet spots are — and why they're architecture-specific. 11 min read · May 28