Song Xiong

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

Cohort 11 of Quantum Open Source Foundation #mentorship program is open for mentee applications until October 26: https://lnkd.in/garJXrh9 I mentored projects as part of this program for several years now (and planning to mentor again this time too), and I think it's a fantastic opportunity for students who already had their introduction to #quantumcomputing and are looking to do a real project in research or #opensource software development for it! Some advice for potential mentees: - Once you've filled the application, you'll have a couple of weeks to solve one of the screening tasks. This is incredibly important! This allows you to showcase your skills and your commitment to the program. There are several times more applicants than applicants who solved a screening task, and the mentors are very unlikely to select someone who didn't complete screening. - Upload your resume as part of your application. Sometimes mentors start off with a project idea in mind, so mentioning your experience with a specific technology or programming language, classical or quantum, can be a plus! - When a mentor reaches out to you, let them know whether you're interested as soon as possible. Quick decisions make everybody's lives easier!

36

⚡ Exciting to see our Block-Max Pruning (BMP) technique in Infinity, an open-source AI-native database designed for LLM applications! In their latest VLDB paper, “Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Search”, Hai Jin, Yingfeng Zhang, and co-authors present a rigorous evaluation of hybrid search architectures — combining full-text, sparse, dense, and tensor retrieval. To support efficient sparse vector search at scale, they’ve integrated BMP into Infinity’s SVS engine — a nice validation of our work on fast, top-k lexical retrieval. 🔗 BMP paper: https://lnkd.in/dsc33hGc 🔗 BMP code: https://lnkd.in/dxBxv225 🔗 Infinity: https://lnkd.in/ddRK5mbr 🔗 Hybrid Search paper: https://lnkd.in/dfBuDXmt Great to see ideas from traditional IR continuing to shape the next generation of retrieval infrastructure!

31

We’re excited to welcome Ray to the PyTorch Foundation 👋 Ray is an open source distributed computing framework for AI workloads, including data processing, model training and inference at scale. By contributing Ray to the PyTorch Foundation, Anyscale reinforces its commitment to open governance and long-term sustainability for Ray and open source AI.    “The PyTorch Foundation is committed to fostering an open, interoperable, and production-ready AI ecosystem. By bringing Ray under the PyTorch Foundation umbrella, alongside projects like vLLM and DeepSpeed, we are uniting the critical components needed to build next-generation AI systems. Ray’s inclusion strengthens our collective mission to support developers with the tools to efficiently train, serve, and deploy AI models at scale.” - Matt White GM of AI at The Linux Foundation Foundation and Executive Director of the PyTorch Foundation “With Ray, our goal is to make distributed computing as straightforward as writing Python code. Joining the PyTorch Foundation helps us stay true to that mission, ensuring Ray continues to be an open, community-driven backbone for developers and their organizations.” - Robert Nishihara, co-founder of Anyscale ➡️ Read the announcement: https://lnkd.in/e3e4mjds #Ray #PyTorchCon #PyTorch #AI #OpenSource #AICompute #DistributedComputing #AIWorkloads #DataProcessing #ModelTraining #Inference

668

5 Comments

🚀 New workshop recording: Reinforcement Learning for Orbital Transfers (Brown University Physics AI Winter School 2026) The Brown University Department of Physics / Center for the Fundamental Physics of the Universe just posted the public recordings from their 2026 AI Winter School, including my 2.5‑hour hands-on module on reinforcement learning (RL) for orbital transfers. In the session we: ▪️ Used Hohmann transfer as an analytic benchmark (minimum‑Δv two‑burn transfer under ideal assumptions) ▪️ Formulated the task as an RL problem (state / action / reward / termination) ▪️ Trained and debugged policies (discrete + continuous thrust), and analyzed classic failure modes ▪️ Compared learned trajectories vs. the analytic baseline using Δv efficiency + stability diagnostics This work bridges physics intuition, modern RL, and the practical workflow of problem framing + debugging. 🙏 Huge thanks to the Brown organizers for inviting me for a second year in a row, especially Ian Dell'Antonio, Rick Gaitskell, Ariel Green, and Chongwen Lu. If you’re curious, the recording, slides, and code (notebook) are now public (link in comments). #ReinforcementLearning #RL #BrownUniversity #Physics #AI

51

3 Comments

"𝐏𝐫𝐨𝐠𝐫𝐚𝐦𝐦𝐢𝐧𝐠 𝐢𝐬 𝐛𝐞𝐢𝐧𝐠 𝐫𝐞𝐟𝐚𝐜𝐭𝐨𝐫𝐞𝐝." — Andrej Karpathy (Former Director of 𝐀𝐈 @ Tesla · Founding team @ OpenAI · CS231n / PhD @ Stanford) This line captures what many developers are feeling but haven't articulated yet. The role of a developer is shifting — 𝐟𝐫𝐨𝐦 𝐭𝐲𝐩𝐢𝐬𝐭 𝐭𝐨 𝐨𝐫𝐜𝐡𝐞𝐬𝐭𝐫𝐚𝐭𝐨𝐫. We're no longer valued only for how fast we write code. We're increasingly valued for how well we design systems of intelligence. Agents. Sub-agents. Prompts, memory, tools, permissions, workflows. This is a new programmable layer — and it's changing everything. Now connect this with a real-world proof point: In 2023, Thomson Reuters acquired 𝐂𝐚𝐬𝐞𝐭𝐞𝐱𝐭 𝐟𝐨𝐫 $𝟔𝟓𝟎𝐌. Not for generic AI — but for 𝐞𝐧𝐜𝐨𝐝𝐞𝐝 𝐥𝐞𝐠𝐚𝐥 𝐞𝐱𝐩𝐞𝐫𝐭𝐢𝐬𝐞. Their product, 𝐶𝑜𝐶𝑜𝑢𝑛𝑠𝑒𝑙, could: • Review legal documents • Research case law • Draft memos • Pass complex legal evaluations with ~97% accuracy They didn't buy "AI". They bought a 𝐃𝐢𝐠𝐢𝐭𝐚𝐥 𝐅𝐓𝐄 — expertise captured, systematized, and deployed 24/7. This is the 𝐀𝐠𝐞𝐧𝐭 𝐅𝐚𝐜𝐭𝐨𝐫𝐲 idea in practice: 𝐃𝐨𝐦𝐚𝐢𝐧 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 → 𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 → 𝐀𝐠𝐞𝐧𝐭𝐬 → 𝐃𝐢𝐠𝐢𝐭𝐚𝐥 𝐅𝐓𝐄𝐬 A Digital FTE isn't hype. It's a purpose-built agent that: • Works continuously • Executes a workflow reliably • Encodes rules, judgment, and constraints • Scales without linear hiring This explains Karpathy's deeper warning: ➤ Failing to claim the AI leverage is now a skill issue. The new skills aren't just frameworks or libraries. They are: • Specification-driven thinking • Agent orchestration • Tool & memory integration • Testing and evaluation for reliability • Turning knowledge into systems that do real work In this paradigm: • Code is no longer the product • 𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 𝐚𝐫𝐞 𝐭𝐡𝐞 𝐩𝐫𝐨𝐝𝐮𝐜𝐭 • Developers become conductors • Agents become the orchestra This shift isn't documented in textbooks yet. Best practices are emerging in production, not tutorials. Those who learn by building will define the standards. The real question isn't: "Will AI change software development?" It's: 𝐖𝐢𝐥𝐥 𝐲𝐨𝐮 𝐛𝐞 𝐛𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐃𝐢𝐠𝐢𝐭𝐚𝐥 𝐅𝐓𝐄𝐬 — 𝐨𝐫 𝐜𝐨𝐦𝐩𝐞𝐭𝐢𝐧𝐠 𝐚𝐠𝐚𝐢𝐧𝐬𝐭 𝐭𝐡𝐞𝐦? #AgenticAI #AIAgents #DigitalFTE #AIEngineering #FutureOfProgramming  #SoftwareEngineering #SpecDrivenDevelopment #LLM #LearningInPublic

18

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments

It’s interesting to see the shift around reinforcement learning — it’s no longer about whether RL is the right approach, but how we can train it more efficiently and at scale. I’m expecting to see a lot more domain-specific applications pop up, both in public and private (enterprise) spaces. https://lnkd.in/gjc3RdcP

18

1 Comment

Audrey Cheng ,Shu Liu and Melissa Pan from University of California, Berkeley gave a wonderful talk at Meta on how AI disrupts system research. Very eye-opening results showing AI outperforming the state of the art in system performance and unlocking new research opportunities!

83

5 Comments

One of the best overviews I read on how Agent Dev Kit (ADK) integrates with Model Context Protocol (MCP) is this new article from Kaz Sato https://lnkd.in/gHRwU3xw Summary of the takeaways that stood out: ⚙️ ADK makes it straightforward to consume MCP servers via MCPToolset utility - it handles connections, discovery, schema adaptation, so external MCP servers "look native" inside your ADK agent. ⚙️ ADK simplifies 2 integration patterns: 🤖 Make your agent an MCP client 🧰 Expose your tools via an MCP server, so other agents can consume your tool via MCP ⚙️ For both patterns communication is async which makes a lot of sense in real-worlf workflows - non-blocking, stateful, multi-turn ⚙️ And if you're building agents that interact with databases, MCP Toolbox for databases is a fast path to bring database access as standardized MCP tools, without having to custom-build these connectors yourself. Overall the more I learn about MCP, it is awesome that a solid standard emerged so early that brings consistency to this growing ecosystem saving overhead of building extra connecting layers.

13