I'm an inference-systems engineer testing efficiency papers and tuning folklore on hardware from a 16 GB MacBook to rented datacenter GPUs. My work asks what actually limits LLM inference — and how those limits change across scales. I drive iterative problem-solving, with coding agents handling lookup, verification, and execution.
KV Cache Reality Lab — kv-cache-tax: one model, one harness, cross-hardware (M3 vs rented NVIDIA L4), four measured legs:
- The KV-cache tax — what long context costs; includes a proven thermal-throttle finding on fanless Apple Silicon.
- The eviction tax — a rotating KV window saves 36% of memory and collapses recall 5/5→0/5 the moment context crosses the window.
- KV quantization: 4× slower on my Mac, 28% faster on an L4 — the same knob, opposite results; whether it helps is a property of the implementation × hardware pair, not the technique.
- Every number measured, medians of interleaved passes, raw data and rental runbooks in the repo.
Inference cost engineering
- A measured audit of Llama-3.1-8B on a rented L4: FP8 cuts cost from $1.00 to $0.36 per million output tokens; everything else I tested — including the tuning-guide favorites — added nothing on top.
Open source (as robertlangdonn)
- mlx-lm: merged #1349 (Gemma 4 unified checkpoint loading); #1329 approved (tekken-v13 detokenizer root-cause, writeup); #1552 (deepseek_v32 sparse-attention indexer fix) in review.
- villa (Vesuvius Challenge): merged #1177 (zarr 3 multiscale read fix) and #1373 (opt-in chunk cache); awarded a $1,000 Progress Prize.
- vLLM #49050 (in review): measured FP8-KV-cache performance documentation, backed by the Reality Lab data.
- llm-compressor #2938 (merged): Cohere2MoE + DeepseekV2 quantization mappings.
- vllm-metal #382 (merged).
From scratch, honestly benchmarked
- mlx-metal-kernels — GPU kernels for LLM inference written from scratch in Metal on an M3; five verified rungs.
- ondevice-bench — execution-checked local-LLM benchmarks (no rubric scoring), including three bugs in my own benchmark.
- lora-toolcalls — LoRA tool-calling fine-tune on a MacBook, with the adapter's inference cost measured, not assumed.
Professionally: production LLM pipelines (output quality gates, hallucination catching, idempotent webhook-driven retries) and end-to-end automation architecture. Previously: paperd.ink, an open-source ESP32 e-paper board used by makers across 20+ countries, and vcfkit, a Rust genomics CLI 4× faster than bcftools.
Writing at prasadkhake.com · On Device