Doug Burger

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Happy to share our recent work on how to efficiently extend context length capabilities of LLMs - https://lnkd.in/gvHamtaE Along with the recipe, we also are open sourcing our synthetically generated datasets which helped us achieve high accuracy on LongBench and HELMET. Link - https://lnkd.in/g7yfDUn8. What’s next - supporting infinite context at inference time for natural language tasks like QA and summarization using these models.

122

2 Comments

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

Today, Microsoft joined Google's developed Agent2Agent (A2A) framework ecosystem to contribute to, and advance the protocol, including interoperability with Azure AI Foundry and Copilot Studio. This is good news for the entire ecosystem! Next to MSFT, we have already welcomed over 50 major Tech players who, together, are shaping the world of AI Agents. Onwards and upwards 🚀

562

13 Comments

💥JUST IN: NVIDIA + Dell Technologies + U.S. Department of Energy (DOE) + Berkeley Lab announce Doudna Supercomputer named after Nobel laureate Jennifer Doudna (pictured on the left)! This system based on our next-generation Vera Rubin (CPU + GPU) architecture will serve a community of 11,000+ scientists and researches in the DoE ecosystem. Official press release: https://lnkd.in/eaPz-ctc Dion Harris NVIDIA Data Center

1,674

24 Comments

I'm happy to share that we've released Instella-Math: a reasoning-focused language model trained with long CoT reinforcement learning on AMD GPUs. It is fully open – its base model (Instella-3B-Instruct), reasoning SFT, and RL training are all fully open. In addition to commonly used math and science reasoning benchmarks, we also evaluate on a new benchmark, TTT-Bench, targeting strategic, spatial, and logical reasoning. Our model performs surprisingly well on TTT-Bench without any exposure to similar-style data during any stage of training. #AMD #LLM #ML #AI #HW #MI300X https://lnkd.in/gnkFp7uQ

206

11 Comments

10/25: Tenth in the series on 25 Google papers for 25 years of WSC: Google-Wide Profiling The "Always On" Monitor: Most companies turn profiling off in production because they fear the overhead. At Google, we keep it on. This paper (Google-Wide Profiling) explains how to get continuous, fleet-wide visibility with a tiny tax: just 0.01% overhead. Why bother, you might ask? Because "lab" environments can be misleading. By profiling in prod, you find: Hardware differences you didn't expect; "Cold" code that is actually running hot; Library bottlenecks across every binary in the fleet. It’s the difference between guessing where the performance went and _knowing_. And for today's nano-banana poster for the paper, I tried a different prompt: [Make an infographic academic-style poster for the following paper. Use a visual aesthetic similar to the architectural blueprint roll -- Large blue sheets with white technical lines, measurements, and room labels.]

378

12 Comments

Want to know how Google DeepMind scales Gemini inference? Rahul Arya, ranked #82 on the Metis List, is giving an invited talk at the ML for Systems workshop at NeurIPS! He will be discussing Trillium, Ironwood, and trade-offs to achieve new speeds in LLM serving. https://mlforsystems.org/ Title: Advances in LLM Serving Efficiency at Scale Bio: Rahul Arya is a research engineer at Google DeepMind contributing to the training and inference performance of Gemini models. He previously worked on the XLA:TPU compiler.

338

3 Comments

New work on optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a new perspective on spending test-time compute: Paper + Code: https://lnkd.in/eUfzZSaz This viewpoint allows us to interpret the long output stream from an LLM as a series of test-time episodes, leading us to use a notion of cumulative regret over output tokens as a metric for measuring test-time compute efficiency. Similar to how RL algorithms can balance exploration and exploitation over training, minimizing cumulative regret can also provide the best tradeoff between exploration and exploitation in the token stream. Using these insights, we develop Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL. Joint work with Yuxiao Qu, Matthew Yang, Amrith Setlur, Lewis Tunstall, Edward Beeching, Aviral Kumar

394

5 Comments