Si Lao

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

We’re excited to welcome Ray to the PyTorch Foundation 👋 Ray is an open source distributed computing framework for AI workloads, including data processing, model training and inference at scale. By contributing Ray to the PyTorch Foundation, Anyscale reinforces its commitment to open governance and long-term sustainability for Ray and open source AI.    “The PyTorch Foundation is committed to fostering an open, interoperable, and production-ready AI ecosystem. By bringing Ray under the PyTorch Foundation umbrella, alongside projects like vLLM and DeepSpeed, we are uniting the critical components needed to build next-generation AI systems. Ray’s inclusion strengthens our collective mission to support developers with the tools to efficiently train, serve, and deploy AI models at scale.” - Matt White GM of AI at The Linux Foundation Foundation and Executive Director of the PyTorch Foundation “With Ray, our goal is to make distributed computing as straightforward as writing Python code. Joining the PyTorch Foundation helps us stay true to that mission, ensuring Ray continues to be an open, community-driven backbone for developers and their organizations.” - Robert Nishihara, co-founder of Anyscale ➡️ Read the announcement: https://lnkd.in/e3e4mjds #Ray #PyTorchCon #PyTorch #AI #OpenSource #AICompute #DistributedComputing #AIWorkloads #DataProcessing #ModelTraining #Inference

668

5 Comments

New from O’Reilly + Redis: Managing Memory for AI Agents 📘 AI agents are only as good as their memory. This new report explores how short-term, long-term, and persistent memory shape agent performance—and how Redis provides real-time, scalable architectures for production-ready AI. 📖 Read the full report: https://lnkd.in/g7t--57p

50

1 Comment

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

Closed stacks may feel convenient, until something breaks. Then you inherit opacity, vendor lock-in, and limited recourse. At scale, open source with Pytorch delivers reproducibility, transparent debugging, and true workload portability as requirements evolve. Open source isn’t ideology. it’s risk mitigation for training infrastructure. Read more:https://lnkd.in/eyFxraSY

19

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

this is super important because TPUs have definitely advantages for cost and performance over other accelators but were much more difficult to use. no longer the case. "vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path. It is not only faster than the previous generation of vLLM TPU, but also offers broader model coverage and feature support." https://lnkd.in/diDKzYbS

35

1 Comment

Excited to share a new top validated submission to #DABstep: Aeya from Tsinghua University Instead of a pure coding agent in a ReAct loop, Aeya’s agent builds relational query plans from a set of relational & _semantic_ operators — then executes them with an engine. The use of semantic operators bridges the gap between the natural language domain to the relational domain and once you get to the relational domain, you don't need arbitrary code generation -- this seems to reduce failure cases significantly!

36

5 Comments

Make a list, and check it twice (automatically) 📜✅ In this #DEVcember demo, Qingyue(Annie) Wang shows Stephanie Wong how to turn manual agent testing into a reliable, automated workflow using pytest. See the workflow → https://goo.gle/4arZLJ6

252

5 Comments

Model training on TPUs just got way easier! A completely reimagined vLLM TPU for LLM inference 👉 for PyTorch and JAX developers, this means more flexibility to run PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX. read: https://lnkd.in/gW7476Hs #GoogleCloud #TPUs #LLMs

14

At Amazon's AGI Lab, XJ Wang's Reinforcement Learning Pod team trains AI agents using RL gyms for real-world applications. His team handles everything from model training to product integration across Amazon. Learn more:

65

2 Comments

⚡ Exciting to see our Block-Max Pruning (BMP) technique in Infinity, an open-source AI-native database designed for LLM applications! In their latest VLDB paper, “Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Search”, Hai Jin, Yingfeng Zhang, and co-authors present a rigorous evaluation of hybrid search architectures — combining full-text, sparse, dense, and tensor retrieval. To support efficient sparse vector search at scale, they’ve integrated BMP into Infinity’s SVS engine — a nice validation of our work on fast, top-k lexical retrieval. 🔗 BMP paper: https://lnkd.in/dsc33hGc 🔗 BMP code: https://lnkd.in/dxBxv225 🔗 Infinity: https://lnkd.in/ddRK5mbr 🔗 Hybrid Search paper: https://lnkd.in/dfBuDXmt Great to see ideas from traditional IR continuing to shape the next generation of retrieval infrastructure!

31

We had great papers and demos on 12/9 at #ASRU. Here are some of them: 👉 Selection of Layers from Self-supervised Learning Models for Predicting Mean-Opinion-Score of Speech, Xinyu Liang, Fredrik Cumlin, Victor Ungureanu, Chandan K. A. Reddy, Christian Schuldt, Saikat Chatterjee. Investigate which layers of self-supervised speech models are most effective for speech quality assessment. Early layers of self-supervised models are better than final layers for predicting speech quality (MOS), leading to simpler, superior SQA models. 👉Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models Tolúlopé Ògúnrèmí, Christopher Manning, Dan Jurafsky, Karen Livescu. Analysis shows Whisper-based SLMs form a semantic interlingua, while ASR-only encoders yield phonetic transliterations, revealing different audio-to-text linking methods. 👉Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data, Gokul Karthik Kumar, Rishabh Saraf, Ludovick Lepauloux, Abdul Muneer, Billel Mokeddem, Hakim Hacid. Falcon3-Audio, trained end-to-end in a single stage on fewer than 30K hours of fully public audio. The 7B model matches state-of-the-art open-weight performance on the MMAU benchmark, suggesting that complex training curricula and multi-encoder architectures are not necessary for competitive audio understanding. Also, the conversational AI demo by Tatiana Likhomanenko was amazing!!

31

1 Comment

How do you scale inference and RL on petabytes of autonomy data 🤖? At Ray Summit 2025, Yi Sheng Ong and Eric Higgins detailed how Applied Intuition leverages Ray on K8s to orchestrate inference and reinforcement learning at petabyte scale . Full talk ➡️ https://lnkd.in/grsQfNpr

143

2 Comments

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Let's learn more! In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point—MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable—without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. I created it while taking inspiration from Femke Plantinga's post on this topic. If you want to understand this better with code, I have linked my article in the comments. ____ If you want to learn AI/ML engineering, I have put together a free PDF (530+ pages) with 150+ core DS/ML lessons. Get here: https://lnkd.in/gi6xKmDc ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

1,384

48 Comments

We’ve worked with Heroku to make Pydantic AI integrate seamlessly with Heroku Managed Inference & Agents. This combination lets you: * Build agents in a familiar Pythonic style, with Pydantic’s data validation at the core * Deploy and run them on Heroku’s fully managed infrastructure * Use standard protocols like Model Context Protocol (MCP) and Agent-to-Agent (A2A) for interoperability The blog post walks through creating and deploying a Pydantic AI agent on Heroku in just a few steps. Read the guide here: https://lnkd.in/epwR98kH

67

2 Comments