Jithu Joijoide

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://lnkd.in/ga_qJkhC

7

Did you hear it? … the song of the AI Agents dominating #awsreinvent2025 Nearly every other booth you walk to is demoing an #aiagent platform. In a nutshell, we’re talking about simple ai agents. I’d call them baby agents. A prompt, a RAG, and a few MCPs here and there to collect data, all governed by an agent studio to design agent workflows. Most platforms are promoting an open model framework where you can pick and choose your preferred LLM. Bedrock is one of the key options available. What’s missing? The business application. A lot of those platforms are in their very early stages, many are not even GA yet. What’s common: build from scratch mentality, and my ai agent platform can do everything, everywhere, all at once. IT buyers? Torn down between all of these platforms not knowing if it’s the right time to start choosing one or many. The potential? It’s going to be insanely huge, with businesses and organizations reaching incredible levels of automation, while also providing superior quality and experience to their customers. Get ready as 2026 will be exciting! [sounds surreal to say so, nearly have to apologize that my post did not use any single AI, not even for brainstorming or improving my style. WDYT: Better with or without AI?]

16

4 Comments

I’ve been experimenting with structured knowledge graphs, optimizing semantic search, and distilling reasoning frameworks that sit on top of LLM outputs. Despite all the progress, no one has yet solved how large language models can offer meaningful guidance about the future. The real breakthrough may come not from bigger models, but from architectures that build persistent agentic memory, systems that learn, recall, and reason more like the human brain. DM me if you see something worth a deeper conversation.

51

Learn how to build an image search app using a series of guided code commits with Developer Advocate Guy Royse. He’ll guide you through generating image embeddings using freely available Hugging Face models and storing them in Redis for fast, scalable semantic search.  The workshop covers: ▶️ What embeddings are and how they work ▶️ How to generate image embeddings and store them in Redis ▶️ Building a semantic search app with Redis, Python, and JavaScript Get access here: https://lnkd.in/gbZ3imbz

44

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

AI works best with a human in the loop 🔁. In this episode, Ion Stoica, Co-founder of Anyscale and Databricks, explains why the most successful AI applications like code assistants and customer support thrive when humans are still reviewing and approving the output. He breaks down how this model applies to data teams using tools like Coalesce: boosting productivity without sacrificing accuracy or control. 🎥 Watch the full recording to hear Ion’s take on AI, reliability, and the future of data workflows: https://lnkd.in/gWzQRggW

19

I'm happy to share that we've released Instella-Math: a reasoning-focused language model trained with long CoT reinforcement learning on AMD GPUs. It is fully open – its base model (Instella-3B-Instruct), reasoning SFT, and RL training are all fully open. In addition to commonly used math and science reasoning benchmarks, we also evaluate on a new benchmark, TTT-Bench, targeting strategic, spatial, and logical reasoning. Our model performs surprisingly well on TTT-Bench without any exposure to similar-style data during any stage of training. #AMD #LLM #ML #AI #HW #MI300X https://lnkd.in/gnkFp7uQ

206

11 Comments

FlagGems Joins the PyTorch Ecosystem: Triton-Powered Operator Library for Universal AI Acceleration In the race to accelerate large language models across diverse AI hardware, FlagGems delivers a high-performance, flexible, and scalable solution. Built on Triton language, FlagGems is a plugin-based PyTorch operator and kernel library designed to democratize AI compute. 💡Learn more and read our latest blog: https://hubs.la/Q03tDbLR0 #FlagGems #PyTorch #PyTorchEcosystem #Triton #AIAcceleration

73

1 Comment

We compared self-hosted and API-based LLM deployments across a range of models, including quantized variants. One of the key factors that emerged was the role of utilization in determining cost-efficiency. Even with FP8 quantization and tuned hardware, the self-hosted environments didn’t come close to matching API efficiency. The gap wasn’t due to performance—latency and throughput were strong when the systems were busy. The challenge was keeping the GPUs consistently utilized. When utilization dropped, unit costs increased significantly. The API-based models were generally stable over the 7-day window. However, we did observe some short periods of degraded performance—spikes in latency and changes in throughput. These weren’t persistent, but they highlight a potential challenge in scenarios where predictable response times are required. Overall, the testing reinforced that deployment tradeoffs depend heavily on workload characteristics. Sustained, high-throughput demand makes it easier to justify self-hosting. For more variable or bursty workloads, efficiency is harder to maintain without advanced orchestration. Full research note is available here: https://lnkd.in/eeCY2Z5R

32

The era of "One Model" is over. NVIDIA Nemotron-3 just validated the specialized "System of Agents" architecture. 🏗️ As AI Architects, we constantly balance the trade-off between massive reasoning models (like o1) and high-throughput operational models. The new NVIDIA Nemotron-3 family is a direct response to this architectural need. They aren't just releasing "another LLM." They are releasing a hierarchical stack designed for multi-agent systems: Nemotron-3 Nano: A 30B parameter model with a Hybrid Mamba-Transformer MoE architecture. Why it matters: It uses Mamba layers for linear-time context processing (handling 1M+ tokens efficiently) and Transformer layers for reasoning attention. This is the perfect "Worker Node" for high-volume agentic tasks. Nemotron-3 Super & Ultra: Larger reasoning engines (coming soon) optimized for orchestration and complex planning. Latent MoE: A novel architecture that projects tokens into a lower-dimensional latent space before routing to experts, significantly reducing the communication bottleneck in large MoEs. The Architectural Shift: This validates the move towards Disaggregated Agent Architectures. Instead of forcing one giant model to do everything, we use specialized, high-throughput "Nanos" for 80% of the work (RAG, summarization, tool calls) and reserve "Ultras" for the 20% deep reasoning. Also, they open-sourced the NeMo Gym RL environments. We finally have a standardized way to train agents on interaction, not just next-token prediction. If you are building agent swarms, the Nano model's throughput/cost ratio is something you need to benchmark immediately. 👇 Link to the blog in the comments. #AIArchitecture #Nemotron3 #NVIDIA #LLM #AgenticAI #SystemDesign #Mamba #MoE

16

1 Comment

DGX Spark seems like a cool product that simplifies AI dev such that debugging things can be easier. I.e. being able to look at the generated videos from ablation experiments without needing an scp. One important question is what is the time to compile PyTorch from scratch though? DGX Spark only has 20 tiny ARM cores (with 10 out of the 20 being even smaller ARM cores). It already takes a decent chunk of time compiling on dual socket xeon/eypc and dual socket DC grade grace CPUs.

27

Building a Large Language Model (LLM) isn’t just about massive compute or model architecture; it’s about the data journey that gets it there. At Cogito Tech, we support this journey end-to-end, from curating domain-specific datasets and aligning transformer-based architectures to fine-tuning, evaluation, and continuous improvement. Every stage — data collection, annotation, and human-in-the-loop validation enhances precision and context that take models from prototype to production, faster. Here’s a quick snapshot of how we help build smarter, robust and domain-aware LLMs. https://lnkd.in/ef6DtJra #HumanInTheLoop #FineTuning #LLM #AI #DataCollection #DataAnnotation #GenerativeAI #ModelTraining #CogitoTech

19

Just getting started with LLMs? Don’t let the jargon slow you down. This 5-part LLM Glossary is your shortcut to mastering the fundamentals👇 1️⃣ Model Types • Foundation Model: Pretrained on large-scale, diverse datasets to learn general capabilities. • Instruction-Tuned Model: Further trained to follow user instructions precisely. • Multi-modal Model: Handles multiple input/output formats like text, images, and audio. • Reasoning Model: Optimized for logic, problem-solving, and step-by-step thinking. • Small Language Model (SLM): Lightweight models for fast, efficient, on-device tasks. 2️⃣ Training & Fine-Tuning • Pretraining: Initial learning phase using massive, diverse data sources. • RLHF (Reinforcement Learning with Human Feedback): Aligns model behavior with human preferences. • DPO (Direct Preference Optimization): Uses ranked preferences instead of rewards for training. • Synthetic Data: AI-generated data used to supplement real datasets. • Fine-Tuning: Targeted retraining on specific domains or tasks. • LoRA / QLoRA: Techniques for efficient low-resource fine-tuning. • Guardrails: Rules or filters applied to enforce safety, ethics, and compliance. 3️⃣ Prompt Engineering • System/User Prompts: Define model behavior and user context in a session. • Chain-of-Thought (CoT): Prompts that guide the model to reason step-by-step. • Few-Shot / Zero-Shot Learning: Demonstrating tasks with few or no examples. • Prompt Tuning: Optimizing prompts via training for specific outcomes. • Context Window: The total number of tokens the model can "remember" per interaction. 4️⃣ Inference & Generation • Temperature: Adjusts output randomness; lower values are more deterministic. • Max Tokens: Limits the number of tokens in a generated response. • Seed: Controls reproducibility of outputs. • Latency: Time taken by the model to respond. • Hallucination: When the model generates factually incorrect but plausible-sounding information. 5️⃣ Retrieval-Augmented Generation (RAG) • Retrieval: Fetching relevant data from external sources before generation. • Semantic Search: Search based on meaning, not just keywords. • Embeddings: Vector representations of text used for similarity matching. • Chunks: Dividing documents into smaller segments for better retrieval. • Vector Databases (VectorDBs): Stores embeddings for fast and accurate search. • Reranking: Rescoring retrieved documents to prioritize relevance. • Indexing: Structuring data for efficient retrieval during generation. 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources) channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

232

36 Comments