Siddesh Shinde

*** Cross-die links introduce skew, coherence issues, and latency that don’t appear in monolithic SoCs. *** Chapter 5 of my book on chiplets details die-to-die interconnects (UCIe, BoW, AIB, and others) and the hooks needed to validate them. Now available on Amazon in hardcover, paperback, and Kindle. Links in the comments.

19

1 Comment

What does acceleration feel like to an engineer? On the left: hours lost to data prep and workflow restarts. On the right: that time reclaimed for modeling, testing, and shipping. Faster hardware doesn’t just cut tasks, it changes the shape of your day. Chapter 4 of The Codex report walks through the math and the culture shift that follows. 🔗https://lnkd.in/gGXhxe78 #theseus #VoltronData

11

1 Comment

Good interview with Jensen Huang and I could not agree more: "You’re not going to lose your job to an AI, but you’re going to lose your job to someone who uses AI." "In today's landscape, understanding how to use Al is no longer a luxury; it's a critical survival skill." "Mark Cuban's investment in free Al bootcamps for underserved high school students exemplifies this philosophy. Cuban urges learners to embrace Al early, absorbing its principles and applications to remain competitive in the evolving job market."

45

5 Comments

LLM inference has become an OS problem. there's working sets, paging, fragmentation, eviction, swap, memory tiers, all happening inside GPU memory. here's how we got here: the KV cache has become the central managed resource. every request claims memory and holds it until completion, exactly like a process claiming pages in RAM. for llama 3-70B at 128K context, one user's KV cache is 40 GB. 8 concurrent users = 320 GB, literally more than the model weights. before vLLM, systems pre-allocated contiguous memory for each request's max sequence length, which led to ~60-80% of KV cache memory being wasted. PagedAttention fixed this by borrowing straight from 1960s OS virtual memory: split the kv cache into fixed-size blocks, map them through a block table, allocate on demand. "think of blocks as pages, tokens as bytes, and requests as processes." waste dropped to under 4%, and throughput jumped 2-4x! but paging was just the start in the OS makeover of inference serving. when HBM fills up, something has to go, and the field reinvented OS page replacement. StreamingLLM found that first tokens absorb disproportionate attention regardless of content. evict them and inference crashes. these are pinned pages -- always resident and never evictable. then came swap. HBM (3.35-8 TB/s) → CPU DRAM (~100 GB/s) → NVMe (~7 GB/s). 50x bandwidth drop per tier. vLLM swaps preempted requests' kv blocks to CPU DRAM. sharing eventually showed up too. SGLang stores kv in a radix tree. requests sharing system prompts point to the same physical blocks. when one diverges, copy-on-write kicks in. this is basically fork() for inference. scheduling maps just as cleanly. prefill is compute-bound, saturating tensor cores. decode is memory-bound, so it loads the entire kv cache for one token. continuous batching = time-sharing. and now it's gone distributed. Mooncake pools kv cache across entire GPU clusters via RDMA at petabyte-scale. kv cache becomes network-attached memory. NVIDIA Dynamo routes requests to where kv data already lives, scheduling meets memory locality. 40 years of OS memory management: paging → eviction → swap → distributed shared memory recreated in 3 years inside HBM. what OS concept will inference systems reinvent next?

587

12 Comments

Cerebras Systems inside of Cline provides lightning fast inference inside your developer workflow. Here is a quick tutorial showing just how easy it is. There are lots of things worth waiting for. Inference results aren't among them. Build with the fastest...

177

Ditto in Action: A Success Story of Real-World Adoption of NVIDIA Torch-TensorRT The latest NVIDIA blog highlights how Torch-TensorRT is enabling fast and efficient LLM inference in production, through the open-source framework Ditto developed by SqueezeBits. By effortlessly converting Hugging Face models into optimized TensorRT engines, Ditto demonstrates the practical value of Torch-TensorRT for accelerating deployment without compromising flexibility or usability. 🔗 GitHub: https://lnkd.in/ge8CQDGp 🔗 Blog: https://lnkd.in/gSUwn-N2

38

1 Comment

Designing AI infrastructure begins with the compute layer, where it's all about striking the right balance between cost, scalability, and performance. It’s equally important to pay close attention to the networking side—since AI workloads often run across multiple nodes, we need fast, low-latency, high-bandwidth connections between them. Thankfully, ongoing innovations and evolving standards are making this more achievable than ever. Given the sheer volume of data involved, the storage layer also plays a critical role. You’ll want high I/O throughput and the ability to access data in parallel, along with solid data versioning. High-performance local storage like NVMe is definitely worth considering as part of the setup. Data preparation is another major piece of the puzzle—especially when you're dealing with large-scale datasets. Automating as much of this process as possible is key to building a system that’s both efficient and scalable. From a model development perspective, the infrastructure should make it seamless to build and train models, ideally with support for distributed training and modular frameworks. Behind the scenes, orchestration tools should handle resource allocation and workflow scheduling smartly, while also supporting secure multi-tenancy—especially in shared environments. Finally, having real-time observability into your infrastructure is a huge advantage. It helps teams monitor system health, spot bottlenecks, and optimize performance on the fly. #AI #Networking #datacenter #RDMA #GPU #Infrastructure #design #AI-STACK #data #AINIC #RNIC

12

We’re working with the OSS community to take the guesswork out of disaggregated serving by integrating NVIDIA Dynamo into the stack, with support for all major inference serving frameworks. 🔹 SGLang community is improving AI inference performance—reducing guesswork and enabling faster, more efficient, and scalable model execution. 🔹 Mooncake built the first SGLang backend for AIConfigurator, enabling rapid support for models like Llama, Qwen, and DeepSeek by implementing the collector layer for core operations such as GEMM and attention. 🔹 Alibaba Cloud integrated AIConfigurator into its AI Serving Stack on Kubernetes (ACK), using the RoleBasedGroup (RBG) orchestration engine to automate deployments and manage prefill/decode disaggregation. The result: 1.86× higher throughput on Qwen3-235B‑FP8 while maintaining TTFT < 5 s and ITL < 40 ms. Read the technical blog → https://lnkd.in/eXhr3vGa

61

9 Comments

You can think of agentic engineering as just another compiler layer, which sits on top of your C++ compiler, which sits on top of LLVM, etc... But the analogy breaks down, because prompts are ephemeral, and don't stick as the literate description (in Don Knuth's sense of "literate programming") of the code. The main reason is that your "compiler," the LLM, is neither hermetic nor deterministic. I do wonder if we're in a wrong local optimum of vibe coding as a result: imagine a world in which you check in your prompts as the source of truth, and the resulting code is what's ephemeral, or a blend of both like in CWEB. This would bring some rigor and sharing capabilities to the exercise of prompting agents, and elevate the entire codebase by one abstraction layer. Another big challenge is that the results of a prompt are non-local, and as a consequence, using them as a linear description of the code is not particularly useful: it's more like a version tree. It is be super interesting to think about what a prompting-aware 'git' would look like, and the role of comments vs. prompts in this environment.

121

14 Comments

Not everyone owns expensive GPUs, but decoding-heavy LLM inference (chat and long-form reasoning) often hits GPU memory limits as the KV cache grows. Our APEX system is a profiling-informed scheduler that dynamically splits inference work between the CPU and GPU. It maximizes overlap during decoding, making hybrid inference more efficient on memory-constrained, low-cost GPUs. APEX enables practical deployments without expensive accelerators. Fantastic work by Jiakun Fan and Xiangchen Li of our PEARL - Performance Engineering for Emerging Architectures Laboratory. The paper will appear at the flagship 40th IEEE IPDPS (International Parallel and Distributed Processing Symposium) in iconic New Orleans! https://lnkd.in/eCKzb_n3 #VTCS #KVCache #LowCostAI #SustainableAI #EdgeAI #IPDPS

227

My latest blog post: Using AI to promote divergent thinking A possible explanation for why we observe such a discrepancy between the productivity gains from AI tools of seasoned senior software engineers versus their less-experienced counterparts: Inexperienced problem solvers have a tendency to get stuck when using AI tools to solve complex problems because they bias towards convergent thinking: narrowing possibilities and looking for a specific solution which fits within their preconceptions. Experienced problem solvers who have adopted AI tools tend to use it to augment a scientific process, which includes building mental models, formulating hypotheses, and performing experiments. In this process, they use AI to accelerate divergent thinking: exploration and discovery of possibilities. (Blog post link in the first comment)

28

12 Comments