Mu (Kevin) Lin

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Our Huxley-Gödel Machine learns to rewrite its own code, estimating its own long-term self-improvement potential. It generalizes on new tasks (SWE-Bench Lite), matching the best officially checked human-engineered agents. With Wenyi Wang, Piotr Piękos, Li Nanbo, Firas Laakom, Yimeng Chen, Mateusz Ostaszewski, Mingchen Zhuge. ArXiv: https://lnkd.in/e3zbgJQe Github: https://lnkd.in/e5_UW2MK

729

34 Comments

Search relevance is the backbone of any RAG system. If the retrieval is bad, the answers are bad — no matter how good the LLM is. We just published a deep dive on how we approach this at Dropbox Dash, and I think the core idea is worth sharing because it applies well beyond our product. The challenge: Dash searches across millions (sometimes billions) of enterprise documents to find the handful that matter for a given query. A relevance model ranks those results, and that model needs training data — lots of it. Human labeling is the gold standard, but it's expensive, slow, and can't touch sensitive customer data. Pure LLM labeling is cheap and fast, but uncalibrated LLMs make systematic mistakes. Our approach: use humans to teach the LLM, then use the LLM to scale. A small set of human-labeled examples calibrates the LLM's relevance judgments. Once the LLM meets quality thresholds, it generates hundreds of thousands of labels that train the production ranking model. The result is a 100x amplification of human labeling effort. A few things that made this work in practice: We focused evaluation on the hardest cases — documents users clicked that the LLM rated low, and documents users skipped that the LLM rated high. That's where the learning happens. We gave the LLM tools to research context before judging. Enterprise search is full of internal acronyms and jargon (at Dropbox, "diet sprite" is a performance management tool, not a soft drink). Without context, even strong models get these wrong consistently. We used DSPy (Community) to programmatically optimize prompts against human judgments, which turned prompt tuning from an art into something measurable and repeatable. The pattern — small human reference set, calibrated LLM evaluation, scaled labeling, continuous monitoring — is generalizable to any domain where you need reliable judgments at scale. Full our post on the Dropbox tech blog: https://lnkd.in/gB_5TECm

68

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

We’re working with the OSS community to take the guesswork out of disaggregated serving by integrating NVIDIA Dynamo into the stack, with support for all major inference serving frameworks. 🔹 SGLang community is improving AI inference performance—reducing guesswork and enabling faster, more efficient, and scalable model execution. 🔹 Mooncake built the first SGLang backend for AIConfigurator, enabling rapid support for models like Llama, Qwen, and DeepSeek by implementing the collector layer for core operations such as GEMM and attention. 🔹 Alibaba Cloud integrated AIConfigurator into its AI Serving Stack on Kubernetes (ACK), using the RoleBasedGroup (RBG) orchestration engine to automate deployments and manage prefill/decode disaggregation. The result: 1.86× higher throughput on Qwen3-235B‑FP8 while maintaining TTFT < 5 s and ITL < 40 ms. Read the technical blog → https://lnkd.in/eXhr3vGa

61

9 Comments

From Behavior to Mechanism: Trust in LLMs Large Language Models are already trusted — embedded in products, workflows, and decisions. What’s less clear is how that trust is justified. We don’t trust them because we fully understand their internals. We trust them because they mostly behave as expected. That gap between behavioral reliability and mechanistic understanding is where the next set of AI risks — and insights — live. What has caught my attention is how Anthropic approaches this problem through interpretability. Most teams evaluate LLMs primarily through behavior — prompts in, responses out. Anthropic treats them more like biological systems, where behavior emerges from internal structures we didn’t explicitly design. The neuroscience analogy fits. Just as neurons form patterns that produce behavior, LLMs activate internal features and circuits that shape responses. Anthropic studies interpretability because they don’t want to rely on behavior alone to trust increasingly powerful models. Their underlying belief is simple: To align powerful AI systems, we must understand the mechanisms that produce behavior — not just the outputs. In practice, interpretability helps: - detect fragile internal shortcuts early - verify that safety mechanisms actually exist - intervene before deployment, not after failure Early LLM deployments offered a useful lesson: behavior alone can be misleading. The same intent, phrased differently, could produce different outcomes. Interpretability asks why. Some labs, like OpenAI, prioritize shaping behavior through feedback, red-teaming, and system safeguards. Anthropic leans earlier — toward understanding internal mechanisms before trust is granted. Both approaches are valid. They simply optimize for different risks. What interpretability ultimately offers isn’t certainty — it offers restraint. It helps us see not just when models succeed, but how they succeed, and where that success may rest on fragile shortcuts. In a world where trust in AI is already granted, understanding becomes less about curiosity — and more about responsibility.

24

In this episode, we discuss ImplicitQA: Going beyond frames towards Implicit Video Reasoning by Sirnam Swetha, Rohit Gupta, Parth Parag Kulkarni, David G Shatwell, Jeffrey A Chan Santiago, Nyle Siddiqui, Joseph Fioresi, Mubarak Shah. The paper introduces ImplicitQA, a new VideoQA benchmark designed to evaluate models on implicit reasoning in creative and cinematic videos, requiring understanding beyond explicit visual cues. It contains 1,000 carefully annotated question-answer pairs from over 320 narrative-driven video clips, emphasizing complex reasoning such as causality and social interactions. Evaluations show current VideoQA models struggle with these challenges, highlighting the need for improved implicit reasoning capabilities in the field.

14

3 Comments

Bad press around autonomous driving is inevitable. Especially when a headline involves a child. Recently, Waymo faced intense scrutiny after one of its vehicles struck a child near a school. The child was thankfully only lightly injured, but the story spread fast, and the reaction was strong. As it should be. What’s getting less attention is the uncomfortable nuance. According to Waymo’s data, the system detected the child immediately, braked hard, and reduced speed significantly before impact. Their internal analysis suggests a typical human driver, even an attentive one, would likely have hit the child at a much higher speed given the same conditions. That does not make the incident acceptable. But it does challenge the way we frame these conversations. We tend to ask: “Did the autonomous system fail?” We rarely ask: “Compared to what baseline?” Human driving sets a very low bar. We just don’t notice it because human errors are normalized. The real question isn’t whether autonomous systems are perfect. They’re not. It’s whether they can consistently make fewer and less severe mistakes than humans, especially in chaotic, high-risk environments like school zones. Public scrutiny is necessary. Transparency is non-negotiable. But progress in safety often looks worse before it looks better, because machine mistakes are visible, logged, and headline-worthy in a way human mistakes never are. If we want safer streets, the comparison has to be honest.

39

13 Comments

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

It’s interesting to see the shift around reinforcement learning — it’s no longer about whether RL is the right approach, but how we can train it more efficiently and at scale. I’m expecting to see a lot more domain-specific applications pop up, both in public and private (enterprise) spaces. https://lnkd.in/gjc3RdcP

18

1 Comment

Post-training of LLMs is increasingly important and RLHF remains a necessary step for an overall great model. Today we are releasing 6 new reward models, including GenRMs and multilingual. These models are used to post-train next *-nemotron models. Collection on HuggingFace: https://lnkd.in/grxW4Sux

187

Out team just posted our latest paper “Evaluating the Promise & Pitfalls of LLMs in Hiring Decisions” on arXiv! We found some exciting results: • Benchmarked leading LLMs (GPT-4o, o3, Claude, Gemini, Llama, DeepSeek) against Eightfold’s “Match Score” model on real-world data. • Evaluated both performance (ROC AUC, PR AUC, F1) and fairness (impact-ratio across gender, race, intersectional groups). • Eightfold’s Match Score beat the best LLM on accuracy (ROC AUC 0.85 vs 0.77) and fairness (min race Impact Ratio 0.957 vs 0.809). • Off-the-shelf LLMs still propagate measurable demographic bias without safeguards. • The trade-off between accuracy and fairness is a false dichotomy: carefully engineered, domain-tuned models like Eightfold’s can achieve both accuracy of hiring and fairness of outcomes. https://lnkd.in/guQ2TAYp #machinelearning #ai #eightfold #arxiv #datascience #bias #fairness #ml #data #genai #llms

37

2 Comments

The most recent Queued Up report from Berkeley Lab is out. Steven Zhang and I are proud to be listed as coauthors on this edition. The Interconnection.fyi team worked closely with Joseph Rand and the rest of the LBNL team to provide the data that powers this year's analysis. Supporting this level of research is core to our mission of bringing transparency to the wholesale energy markets. The 2025 edition (covering data through the end of 2024) highlights some significant shifts in the landscape: Active Capacity: 2024 closed with nearly 2,300 GW of generation and storage seeking interconnection. Changing Mix: Active natural gas capacity increased by 72% year over year, while solar and storage saw slight decreases in total queue volume. The Backlog: 408 GW of capacity already has an executed or draft interconnection agreement but has not yet reached commercial operations. Timelines: For projects built between 2018 and 2024, the median duration from request to operation has doubled compared to the early 2000s. This report is the definitive annual benchmark for the industry and provides a vital baseline as we begin to see the implementation of FERC Order 2023. The analysis in this report is based on our EOY 2024 data snapshot. Since then, we have continued to track and update these queues every single day. If you want a live view of how these numbers have shifted in the months since this snapshot was taken, follow Interconnection.fyi and subscribe to our Substack. You can find the full slide deck, interactive maps, and raw data files at the link in the comments.

41

3 Comments