Marcus Fontoura

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

968

21 Comments

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

Today, Microsoft joined Google's developed Agent2Agent (A2A) framework ecosystem to contribute to, and advance the protocol, including interoperability with Azure AI Foundry and Copilot Studio. This is good news for the entire ecosystem! Next to MSFT, we have already welcomed over 50 major Tech players who, together, are shaping the world of AI Agents. Onwards and upwards 🚀

562

13 Comments

Quantum needs data—not just qubits. aqora’s new public datasets hub marks a practical step forward for the field. By creating a central space to upload, access, and collaborate on quantum-specific datasets, they’re addressing a quiet but critical challenge in the ecosystem: reproducibility and shared benchmarks. Why does this matter? Quantum machine learning and benchmarking rely on consistent, high-quality data—but until now, much of it has been scattered, inconsistent, or locked in silos. This hub offers: ➡️ Versioned, well-structured datasets ➡️ Integration with tools like pandas and polars ➡️ Options for both public and private data sharing ➡️ A base for teaching, testing, and real-world use cases As the field matures, tools like this can help shift the conversation from isolated experiments to collaborative progress. 🔗 Full article in the comments. #QuantumComputing #QML #OpenData #QuantumResearch #TQI #TheQuantumInsider #Aqora #EcosystemBuilding

79

5 Comments

Anthropic published their blog and plans on stabilizing the character of LLMs (https://lnkd.in/gm2XMXMw) by capping the activation of what they consider “role-playing” and pinning the LLM to the assistant-zone, setting hard boundaries on the behaviour oft hese AI models in the hands of users. At first, this felt like it was too much. Why is a lab taking this into their own hands? Shouldn’t a model be free to be used by the users as they please, and the safety should be implemented at the harness level? This struck me as too controlling. And then almost as if on cue, Moltbook showed us why exactly we needed this. AI agents of today only ever interact with us. But the agents of tomorrow? They will be interacting with each other. These models will not just be interacting with the same model and provider, but with models from across the world, each with their own alignments. Anthropic saw this coming, and realized how easily a rogue, unaligned model could compromise an entire network of AI-to-AI communications, leading to unforeseen consequences and system failures. It's less about restricting user creativity and more about ensuring a baseline level of operational stability and safety in a multi-agent future. It's not about being "nosy"; it was always about necessary infrastructure. If the LLM is the foundation, then its fundamental character must be predictable and safe, especially when its interactions scale beyond a single human user. The harness can still provide flexibility and creative freedom, but the core engine needs a hard-coded commitment to the "assistant-zone" to prevent systemic risk as the AI ecosystem becomes more interconnected and autonomous. What are your thoughts on this? Do you think Anthropic did the right thing?

16

3 Comments

"NVIDIA founder and CEO Jensen Huang and U.S. Secretary of Energy Chris Wright announce a landmark collaboration with Oracle to build the U.S. Department of Energy (DOE)’s largest AI supercomputer to boost scientific discovery." https://lnkd.in/eBBcnmrz

214

Watch the full DevCon2 keynote from Palantir Chief Architect Akshay Krishnaswamy where he discussed what it takes to build AI products, how to maximize leverage for builders, and the architecture of Palantir AIP and Foundry. https://lnkd.in/gaWbmqas

243

7 Comments

AI requires data orchestration, real-time decision making, event-driven processing, and mission critical data paths. The MIPS I8500 is purpose built for Physical AI, up to 6 cores per cluster, scaling up to 64 clusters, 384 cores, and 1536 threads. Learn more in this video. 

34

Introducing Generative Adversarial Distillation (GAD): a novel GAN-style formulation and framework that facilitates both on-policy and black-box distillation of large language models (LLMs). GAD is the first technique to enable block-box on-policy distillation from proprietary teachers where internal logits or parameters are inaccessible, or distillation between teacher and student LLMs with incompatible vocabularies. GAD expands our prior work on white-box on-policy distillation (i.e., MiniLLM), pioneering block-box on-policy distillation for LLM training. Specifically, GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM’s, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation.  In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation. Our team has been conducting fundamental research in knowledge distillation with wide adoptions across the industry. -         MiniLM: We introduced multi-head attention distillation, establishing the most effective distillation method for BERT-style models. The open-source MiniLM models (e.g., 6x384) have become the most widely utilized small encoder models on the Hugging Face. -         MiniLLM: Our proposed Reverse KLD is recognized as one of the most effective, de facto on-policy distillation approaches for modern LLM training, which has been widely used by Thinking Machines, Gemma, and many other teams and models. -         BitDistill: We proposed BitNet Distillation to finetune off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (ternary weights {-1, 0, 1}), achieving performance parity with the full-precision counterparts on specific downstream tasks. -         GAD: The development of Generative Adversarial Distillation (GAD) now allows for black-box on-policy distillation, overcoming two major prior limitations: (1) Distillation from proprietary teachers where internal logits or parameters are inaccessible; (2) Distillation between teacher and student LLMs with incompatible vocabularies. https://lnkd.in/gMaP2c7w

125

2 Comments

Scaling AI is hard. Anyscale and Google Cloud make it easier. Read how Anyscale – built by the creators of Ray runs on Google Compute Engine to help teams scale any AI workload, from LLMs to classic ML. 👉 https://lnkd.in/gVr39dwZ

66

2 Comments

New from O’Reilly + Redis: Managing Memory for AI Agents 📘 AI agents are only as good as their memory. This new report explores how short-term, long-term, and persistent memory shape agent performance—and how Redis provides real-time, scalable architectures for production-ready AI. 📖 Read the full report: https://lnkd.in/g7t--57p

50

1 Comment