Altan Alpay

The New Stack spotlights the PyTorch Foundation’s evolution into an umbrella foundation and the addition of vLLM and DeepSpeed as newly hosted platform projects. “Since its inception at the Linux Foundation two and a half years ago, the PyTorch Foundation has grown to include over 30 member organizations and 120 ecosystem projects. These newest moves are a natural evolution of its rapid growth and global momentum.” 🧠vLLM: Built for fast, efficient LLM inference with innovations like PagedAttention and deep PyTorch integration. ⚙️ DeepSpeed: A go-to library for scalable training, leveraging technologies like ZeRO and 3D parallelism—all built on PyTorch. 📖Read the full article from Steven J. Vaughan-Nichols: https://hubs.la/Q03lFYd00 #PyTorch #vLLM #DeepSpeed #OpenSourceAI #LLMInference

84

1 Comment

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

Scaling AI is hard. Anyscale and Google Cloud make it easier. Read how Anyscale – built by the creators of Ray runs on Google Compute Engine to help teams scale any AI workload, from LLMs to classic ML. 👉 https://lnkd.in/gVr39dwZ

66

2 Comments

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

We just released Nemotron Speech ASR: 🤖0.6B streaming cache-aware transducer 📉low latency (down to 80ms) 📈high throughput (up to 900 concurrent streams on H100) 🎮adjustable latency-throughput-accuracy trade-off without re-training 🌎English ASR 🔗https://lnkd.in/eFeEQR2F

538

20 Comments

You can throw Claude's entire context window at every single AdCP request and it's still cheaper than the RTB auction infrastructure. RTB layer: 500M req/sec × 4ms = ~2 million CPU-cores worth of compute, just to run the auctions. Latency-critical, deterministic, dumb-but-fast. AdCP layer: 100 req/sec of natural language strategy. Latency-tolerant (seconds, not milliseconds). Smart-but-slow. AdCP sits above the auction layer. It's the interface where a human or agent says "shift 20% of budget from prospecting to retargeting" or "find me audiences similar to my converters but exclude high-carbon inventory" - and then traditional systems execute that at auction speed. It's the difference between: - Trading floor (RTB): Millisecond execution, no reasoning - Portfolio manager (AdCP): Strategic allocation, full reasoning The "allocation not valuation" thesis maps perfectly here. AdCP doesn't price impressions - it allocates capital across strategies. The RTB infrastructure continues to do the high-frequency execution. This also explains why AdCP can be agentic in a way RTB never could be. You have time to reason, search, iterate, ask clarifying questions. An agent can take 30 seconds to parse a campaign brief and construct an optimal allocation - that's an eternity in RTB terms but instant in human workflow terms.

189

23 Comments

Speeding up PyTorch performance usually requires the use of intermediate optimization frameworks, or writing low-level optimized kernels. This can be burdensome, especially when supporting multiple target devices. Taras Sereda led research to evaluate if frontier models could automatically generate low-level kernels from pure PyTorch. Our technique sped up inference on Apple by 1.87X on average! Check out the details here: https://lnkd.in/g4tdEXEi

89

5 Comments

We have a new blog post on using FlexAttention in FlashAttention CuTe DSL! FlexAttention is a popular PyTorch API for implementing many attention variants through custom score modifications ("score mod"), masking and block sparsity ("mask mod"), or both. However, while the original Triton implementation performed well on NVIDIA Ampere GPUs, it significantly lagged behind on Hopper GPUs compared to optimized kernels like FlashAttention-3. On the other hand, although it was possible in principle to add FlexAttention support to the C++ FlashAttention kernels via an EVT-like abstraction, the complexity of CUTLASS C++ and lack of native JIT compilation made this task unnecessarily cumbersome. Porting the FlashAttention kernels over to CuTe DSL therefore presented an ideal opportunity to revisit FlexAttention in FlashAttention. In collaboration with Driss Guessous and Tri Dao, we've now added a simple FlexAttention-style API to FlashAttention CuTe DSL that has minimal performance loss, achieving up to 95% of FA-3 on Hopper, for example. This work is currently available in the official Dao-AILab repo and is also integrated into PyTorch nightly. See the blog for a thorough user's guide on the API: https://lnkd.in/gETPGUzV

394

5 Comments

Use case: scaling videogen with Baseten Inference Stack on Nebius ↔️ Generative video workloads run longer than others, stress GPU memory and magnify the impact of latency. Today’s blog shows how Baseten’s optimized runtime (including topology-aware parallelism) and orchestration operate on our clusters: https://lnkd.in/e23Bj7qc. Since last month, Baseten has been running on Nebius across our cloud regions in the US, Finland and France. #Partnerships #AIInfrastructure #ModelInference #VideoGen

63

“Anthropic recently published a blog post about building a C compiler entirely with Claude . They called it CCC (Claude’s C Compiler) and claimed it could compile the Linux kernel. 100% of the code was written by Claude Opus 4.6, a human only guided the process by writing test cases. That sounded interesting enough to test the claim and benchmark CCC against the industry standard GCC” https://lnkd.in/gUpDEU69

12

3 Comments

New from O’Reilly + Redis: Managing Memory for AI Agents 📘 AI agents are only as good as their memory. This new report explores how short-term, long-term, and persistent memory shape agent performance—and how Redis provides real-time, scalable architectures for production-ready AI. 📖 Read the full report: https://lnkd.in/g7t--57p

50

1 Comment

NVIDIA Nemotron-ultra-253b demonstrated strong performance compared to other open-source models, as tested by the Plurai IntellAgent framework. Together with Amit Bleiweiss, we integrated NIM into the Plurai IntellAgent open-source and ran an evaluation that tested the model against various agentic production environments. Enterprises face critical blockers taking agentic LLM workflows into production. Production demands strict attention to data quality, security, compliance, and day to day reliability. This marks another step in standardizing the LLM benchmarking on real-world agentic enterprise use cases. Learn more about the golden standard of agentic enterprise production: Link: https://lnkd.in/dvahzDAr

54

5 Comments

Voice AI is quietly hitting an inflection point 🔊 NVIDIA just published a great deep dive on scaling speech recognition for real-world voice agents - latency, streaming accuracy, robustness, and why “ASR quality” isn’t just about WER anymore. 👉 https://lnkd.in/enDUKHvG A few things that stood out: - Streaming + low latency is non-negotiable if you want natural conversations - ASR needs to handle interruptions, background noise, accents, and partial utterances, not clean demo audio - Scaling voice agents isn’t just model quality, it’s systems engineering If you want to actually try building or running voice agents instead of just reading about them, Pipecat is a great place to start: 👉 https://pipecat.ai This is the kind of progress that makes voice agents finally feel… practical. Not demos. Not prototypes. Production. Curious to see where the ecosystem lands this year 👀

149

10 Comments

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

�� XGBoost 3.0 now has new capabilities for GPUs: ✅ Train up to 1 TB of data on a single Grace Hopper GPU: Up to 8x faster than CPU with External-Memory Quantile DMatrix. ✅ Faster and lighter: GPU hist/approx methods an additional ~2x speed up and reduced memory use. ✅ Full feature support: External memory now supports categorical features, all objectives, and SHAP. ✅ Distributed mode: Experimental support for out-of-core training across a cluster. Technical deep dive ➡️ https://bit.ly/45JW6TV

8

1 Comment