Reza Zadeh

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

🚨 Two AI Legends Just Said the Same Thing: LLMs Aren't the Path to True Intelligence Richard Sutton (Father of Reinforcement Learning, 2024 Turing Award Winner) and Yann LeCun (Meta's Chief AI Scientist) are both moving beyond LLMs. Here's why this matters for everyone: 🧵 The Problem They Both See 🤔 Current AI chatbots (like ChatGPT) are basically sophisticated copy-cats. They: Mimic what humans would say ✍️ Don't actually understand consequences Can't truly learn from mistakes Have no real goals beyond predicting text Think of it like this: A parrot 🦜 can repeat "the sky is blue" perfectly, but it doesn't understand what sky, blue, or "is" means. What Real Intelligence Needs 🧠 Both experts say we need AI that: 1️⃣ Learns like a child does 👶 Not by copying, but by trying things and seeing what happens 2️⃣ Understands the physical world 🌍 Can predict: "If I push this cup, it will fall" 3️⃣ Has actual goals 🎯 Not just "predict the next word" but "achieve this specific outcome" 4️⃣ Plans ahead ♟️ Like chess players thinking multiple moves ahead, not just reacting The Future They Envision 🔮 Instead of bigger chatbots, imagine AI that: Learns continuously from experience 📈 Updates its understanding in real-time Can actually reason about cause and effect Builds mental models of how things work Why This Matters to You 💡 We're potentially at a turning point. The current AI boom might hit a wall 🧱, and the next breakthrough will come from completely different approaches. For businesses: Don't just bet on scaling current AI - watch for fundamentally new architectures For careers: Understanding real-world interaction and planning might become more valuable than prompt engineering The Bottom Line 📝 Two of AI's most influential thinkers are saying the same thing: We've been building really impressive calculators 🖩 when we should be building systems that actually understand the world. The next phase of AI won't just be ChatGPT-5 or GPT-6. It might be something entirely different. 🚀 What do you think - are we heading toward a major shift in how we approach AI? source: https://lnkd.in/gsHZvNbE https://lnkd.in/gchsKwsE #AI #GenAI #AGI #LLM #

41

3 Comments

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

#TuesdayPaperThoughts Edition 60: The Art of Scaling RL Compute for LLMs This week's #TuesdayPaperThoughts explores "The Art of Scaling Reinforcement Learning Compute for LLMs" from researchers at Meta, The University of Texas at Austin, University of California, Berkeley, and Harvard University. While pre-training has had its moments with predictable scaling laws, RL training has remained more art than science until now. Key Takeaways: 1️⃣ Predictive Scaling Framework: The work introduces sigmoidal compute-performance curves for RL training that separate asymptotic performance (A) from compute efficiency (B). This framework enables extrapolation from smaller-scale runs to predict performance at larger compute budgets. The team validated this with a massive 100,000 GPU-hour run where predictions from just the first 50k hours closely matched final performance. 2️⃣ Not All Recipes Scale Equally: Methods that look promising at small compute budgets can hit lower performance ceilings at scale. The study reveals that design choices like loss type and FP32 precision shift asymptotic performance, while factors such as loss aggregation and normalization primarily modulate compute efficiency. This explains why some widely-adopted methods plateau unexpectedly. 3️⃣ ScaleRL Recipe: Through systematic ablations consuming 400,000+ GPU-hours, the team developed ScaleRL—combining PipelineRL with 8-step off-policyness, truncated importance sampling (CISPO), FP32 logits computation, and adaptive prompt filtering. ScaleRL achieves A=0.61 asymptotic performance, outperforming DeepSeek's GRPO, Qwen's DAPO, and other prevalent methods on both ceiling and efficiency. The timing couldn't be better. With RL compute budgets exploding (10× increase between model generations for o1→o3 and Grok-3→Grok-4), the field desperately needed this systematic approach. Research Credits: Devvrit K., Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit Dhillon, David Brandfonbrener, Rishabh Agrawal Paper Link: In comments

21

1 Comment

I’m excited to attend NeurIPS workshop sessions at San Diego and present two papers authored by the applied research team: 1. GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplace. Mikołaj Sacha et al. - NeurIPS Workshop on Unifying Representations in Neural Models (UniReps) 2. Towards Real-World Evaluation of Agentic Work in Freelance Marketplaces. Mattie Terzolo et al. - NeurIPS Workshop on LLM Evaluation Over the next couple of days, I’ll be sharing insights and some behind-the-scenes perspectives on how we think about these models, how they’re shaping Upwork today, and the opportunities we see ahead. For attendees already onsite at the main conference, please stop by the Upwork booth #722 to learn more about the efforts the ML & AI team is driving - from research to real product impact. Feels great to be again publishing at NeurIPS after a hiatus and looking forward to great conversations and inspiration ahead!

50

2 Comments

Today marks another step toward making Reinforcement Learning (RL) faster, simpler, and scalable for builders harnessing the Essential Cloud for AI. Our support on the development of torchforge - a new PyTorch-native framework - is helping researchers take large-scale RL workloads from experimentation to production. In partnership with Meta and Stanford University, torchforge was validated on CoreWeave’s Cloud Platform proving stable, efficient, and highly performant – ready to power the next generation of RL workloads. torchforge is now supported on SUNK. More in our recent blog: https://hubs.la/Q03PLJC10

136

Excited to share our NeurIPS 2025 Tutorial on How to Build Agents to Generate Kernels for Faster LLMs (and Other Models!) A collaboration across institutions: AMD, Stanford, Google DeepMind, Arm, Nvidia, Meta, Modular AI, UC Irvine, and ML Commons. - If you're an AI researcher, check out the parts of AI compute (e.g. GPU) that affect your inference & training jobs -- from why you should do batch sizes in certain multiples of 2 to how your trillions of matrix multiplications make their way through different types of memory on a chip - Also chat with us about the trend of self-improving LLMs that can write code to make them even faster. Agents -> SFT data -> RL with profiling tools for strong reward signals - Come build with us! It's an exciting future where LLMs can make the next generation of AI research (RL, even more sparsity, hybrid models with SSMs) super fast on any new hardware generation -- on-the-fly in your IDE Sina Rafati, Hao Li, Azalia Mirhoseini, Anna Goldie, Laurence Moroney, Vartika Singh, Mark Saroufim, Chris Lattner, Sitao Huang, David Kanter, Simon Guo, Kesavan Ramakrishnan, Vincent Ouyang, Tim Gianitsos, Nithyashree Manohar, Sharon Zhou, with acknowledgements to the AMD GEAK team Zicheng Liu, Dong Li, Ziqiong Liu, Pratik Prabhanjan Brahma and the AMD ROCm Kernel team ZHAOYI LI and the AMD Omni team Muhammad Awad, Keith Lowery, Cole Ramos and kernel engineers John Tyler, Muhammad Osama and of course the one and only Emad Barsoum. When they're out, will drop... * arxiv paper link * Github repo link

1,298

39 Comments

Yann LeCun was recently covered on X saying... "I'm not interested in LLMs anymore - they're the past. The future is in four more interesting areas: machines that understand the physical world, persistent memory, reasoning, and planning." (https://lnkd.in/gX2JdhCY). Why would he make such a statement? The reason lies in the fact that while massive investment in dollars, compute, energy and innovation continues to grow unabated in AI, there remain a nagging set of persistent questions. These questions pertain to the intersection of the newly evolving world of intelligent machines and the practical, deterministic physical reality in which we all live. To reach AI-full potential we will need to create an orchestrating interface between these intelligent machines and the real-world and a collaborating fabric between the multitude of intelligent machines themselves. The attributes that will define such a reliable and productive network are as follows: 1.) The network itself will need to be intelligent and adaptive; able to handle high levels of complexity at global scale and at the edge with reasoning that allows for real-world disruptions that are endemic to everyday life. 2.) The network will also need to be informed by and anchored to the intent of the people guiding it with the planning capability to extend those human derived objectives into action and follow-through. 3.) The architecture will need to be persistent as the machines will need to be capable of autonomy; knowing what they were doing, what they are doing and what they intend to do. 4.) The machines need to be capable of coordinating amongst themselves. With 5.3 Billion people now on the internet and more than 400 Billion machines the future of the internet will be not just about human-to-machine communication but primarily about the machine-to-machine collaboration. 5.) The intelligence will need to be deterministic. Gen-AI with it's probabilistic logic will only get you so far in orchestrating real-time, real world actions that, by definition, are deterministic. 6.) Finally, an aspect not mentioned by Yann LeCun is the increasing importance of cyber-security in a world in which the Intelligence is on the move and actively engaging with people "in the wild". Welcome to the Paranet; a secure, distributed intelligence, asynchronous compute that leverages non-stochastic AI and persistence to create "the Internet of work" where humans and machines but primarily machines to machines can collaborate securely and reliably at the edge.

15

2 Comments

Customers are asking for AI that goes beyond assistance to true collaboration. Claude Opus 4.5 from Anthropic delivers on that vision: understanding objectives, factoring in constraints, and ready to power complex workflows. Opus 4.5 joins the full set of Claude models now in Microsoft Foundry, alongside models from OpenAI, Mistral AI, Cohere, Meta, and others, so you can easily compare and find the best-fit models for your needs. With Microsoft Foundry, Azure customers get the broadest selection of model providers on any cloud, plus the governance, security, and interoperability to accelerate innovation with confidence. 🔗 https://lnkd.in/gdRxNB_q

107

3 Comments

Introducing Generative Adversarial Distillation (GAD): a novel GAN-style formulation and framework that facilitates both on-policy and black-box distillation of large language models (LLMs). GAD is the first technique to enable block-box on-policy distillation from proprietary teachers where internal logits or parameters are inaccessible, or distillation between teacher and student LLMs with incompatible vocabularies. GAD expands our prior work on white-box on-policy distillation (i.e., MiniLLM), pioneering block-box on-policy distillation for LLM training. Specifically, GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM’s, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation.  In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation. Our team has been conducting fundamental research in knowledge distillation with wide adoptions across the industry. -         MiniLM: We introduced multi-head attention distillation, establishing the most effective distillation method for BERT-style models. The open-source MiniLM models (e.g., 6x384) have become the most widely utilized small encoder models on the Hugging Face. -         MiniLLM: Our proposed Reverse KLD is recognized as one of the most effective, de facto on-policy distillation approaches for modern LLM training, which has been widely used by Thinking Machines, Gemma, and many other teams and models. -         BitDistill: We proposed BitNet Distillation to finetune off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (ternary weights {-1, 0, 1}), achieving performance parity with the full-precision counterparts on specific downstream tasks. -         GAD: The development of Generative Adversarial Distillation (GAD) now allows for black-box on-policy distillation, overcoming two major prior limitations: (1) Distillation from proprietary teachers where internal logits or parameters are inaccessible; (2) Distillation between teacher and student LLMs with incompatible vocabularies. https://lnkd.in/gMaP2c7w

125

2 Comments

Out team just posted our latest paper “Evaluating the Promise & Pitfalls of LLMs in Hiring Decisions” on arXiv! We found some exciting results: • Benchmarked leading LLMs (GPT-4o, o3, Claude, Gemini, Llama, DeepSeek) against Eightfold’s “Match Score” model on real-world data. • Evaluated both performance (ROC AUC, PR AUC, F1) and fairness (impact-ratio across gender, race, intersectional groups). • Eightfold’s Match Score beat the best LLM on accuracy (ROC AUC 0.85 vs 0.77) and fairness (min race Impact Ratio 0.957 vs 0.809). • Off-the-shelf LLMs still propagate measurable demographic bias without safeguards. • The trade-off between accuracy and fairness is a false dichotomy: carefully engineered, domain-tuned models like Eightfold’s can achieve both accuracy of hiring and fairness of outcomes. https://lnkd.in/guQ2TAYp #machinelearning #ai #eightfold #arxiv #datascience #bias #fairness #ml #data #genai #llms

37

2 Comments