Atif Adib

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://lnkd.in/ga_qJkhC

7

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments

Model training on TPUs just got way easier! A completely reimagined vLLM TPU for LLM inference 👉 for PyTorch and JAX developers, this means more flexibility to run PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX. read: https://lnkd.in/gW7476Hs #GoogleCloud #TPUs #LLMs

14

Where are Funds putting their AI-Money? Based on the current roles we're focused on, there's large investment in LLM integration for Quantitative Workflows, Time-Series Forecasting/Simulation and Reinforcement Learning/Agentic Systems for Research Specific focus on NLP within unstructured data and RL environment generation around time-series modelling, backtest, simulation and rebalancing, aswell large-scale research and analytics capability. Roles are a healthy mix of Quantitative, Software, Data and Product, all at Senior IC and all with a clear orientation toward AI/ML commercial experience. All firms are looking for hands-on individuals who can blend an exec presence with a sleeves-up approach. Vast majority of hiring sentiment is focused on ICs, not leadership. Most of the leadership roles are filled. Larger firms still favour financial backgrounds and will wait for lengthy NCs to secure the right talent. Smaller firms are open to buy-side to buy-side shift, but are more aggressively looking at MedTech, Pharma, FAANG or Tech StartUps as their source of exceptional talent. Compensation is through the roof, the battle for talent is intensifying. All very much to play for.

14

7 Comments

Building a Large Language Model (LLM) isn’t just about massive compute or model architecture; it’s about the data journey that gets it there. At Cogito Tech, we support this journey end-to-end, from curating domain-specific datasets and aligning transformer-based architectures to fine-tuning, evaluation, and continuous improvement. Every stage — data collection, annotation, and human-in-the-loop validation enhances precision and context that take models from prototype to production, faster. Here’s a quick snapshot of how we help build smarter, robust and domain-aware LLMs. https://lnkd.in/ef6DtJra #HumanInTheLoop #FineTuning #LLM #AI #DataCollection #DataAnnotation #GenerativeAI #ModelTraining #CogitoTech

19

We’ve worked with Heroku to make Pydantic AI integrate seamlessly with Heroku Managed Inference & Agents. This combination lets you: * Build agents in a familiar Pythonic style, with Pydantic’s data validation at the core * Deploy and run them on Heroku’s fully managed infrastructure * Use standard protocols like Model Context Protocol (MCP) and Agent-to-Agent (A2A) for interoperability The blog post walks through creating and deploying a Pydantic AI agent on Heroku in just a few steps. Read the guide here: https://lnkd.in/epwR98kH

67

2 Comments

Finding it hard to turn a machine learning idea into a production-ready application? At the upcoming #SnowflakeBUILD, Senior Developer Advocate Chanin Nantasenamat will show you how to build your first end-to-end machine learning project from scratch. In this hands-on lab, you’ll learn to build the entire ML lifecycle from absolute zero, built entirely on Snowflake: ‣ Data Preparation: Prepare a fun multi-modal bear dataset, working with both structured and unstructured data. ‣ Build and optimize ML models  ‣ Deploy and serve ML model in an interactive data app with Streamlit ✍️ Register today: https://lnkd.in/gGMyByjw

28

🚀 scikit-learn 1.8 is out 🚀 A big shoutout to the community of contributors who continue to push open-source machine learning forward ❤️ ✨ Key Highlights: ▶️ Expanded Array API support (including PyTorch & CuPy) to run more estimators and metrics on GPUs ▶️ Free-threaded CPython 3.14 support for better multi-threaded performance ▶️ Probability calibration with temperature scaling in CalibratedClassifierCV ▶️ Major efficiency boosts in linear models (Lasso / ElasticNet with gap safe screening) ▶️ Much faster and more robust DecisionTreeRegressor with criterion="absolute_error" ▶️ New manifold.ClassicalMDS implementation for classical multidimensional scaling 🔗 Check the full release highlights: https://lnkd.in/gkXQSbmZ Discover scikit-learn 1.8 and its: 🟢 28 new features 🔵 12 efficiency improvements & 13 enhancements 🟡 9 API changes 🔴 34 fixes 👥 193 contributors (thank you all!) 📖 More details in the release notes: https://lnkd.in/gCrs42se You can upgrade with pip as usual: pip install -U scikit-learn Using conda-forge builds: conda install -c conda-forge scikit-learn #scikitlearn #MachineLearning #opensource #DataScience #Python #ML

1,519

14 Comments

AI/ML developers face tough orchestration challenges, including data pipelines, GPUs, failures, and deployments. Temporal’s code-first approach makes it simpler and faster to build reliable systems. Over 90 AI companies already use Temporal Cloud. Check out our blog post to see why 👉 https://lnkd.in/gKxWwMXc

17

We’re excited to welcome Ray to the PyTorch Foundation 👋 Ray is an open source distributed computing framework for AI workloads, including data processing, model training and inference at scale. By contributing Ray to the PyTorch Foundation, Anyscale reinforces its commitment to open governance and long-term sustainability for Ray and open source AI.    “The PyTorch Foundation is committed to fostering an open, interoperable, and production-ready AI ecosystem. By bringing Ray under the PyTorch Foundation umbrella, alongside projects like vLLM and DeepSpeed, we are uniting the critical components needed to build next-generation AI systems. Ray’s inclusion strengthens our collective mission to support developers with the tools to efficiently train, serve, and deploy AI models at scale.” - Matt White GM of AI at The Linux Foundation Foundation and Executive Director of the PyTorch Foundation “With Ray, our goal is to make distributed computing as straightforward as writing Python code. Joining the PyTorch Foundation helps us stay true to that mission, ensuring Ray continues to be an open, community-driven backbone for developers and their organizations.” - Robert Nishihara, co-founder of Anyscale ➡️ Read the announcement: https://lnkd.in/e3e4mjds #Ray #PyTorchCon #PyTorch #AI #OpenSource #AICompute #DistributedComputing #AIWorkloads #DataProcessing #ModelTraining #Inference

668

5 Comments

We've made Heroku Managed Inference and Agents a whole lot more flexible with our new Standard Plan. Now, you can experiment, fallback, and scale across our entire model catalog without the friction of provisioning new add-ons for every change. One add-on. One API key. Infinite possibilities. 🚀 What’s new in the catalog: ☑️ Claude 4.6 family: Opus and Sonnet support for high-reasoning tasks. ☑️ Expanded open-weight models: DeepSeek v3.2, Kimi K2.5, and more. ☑️ Cohere Embed V4: Advanced vector search for better RAG performance. Update your model name in your code and you're good to go. It’s that simple. Learn more here https://sforce.co/3Ozfpt1 #DevEx #AI #Claude #DeepSeek

16

We made a guide on how to deploy LLMs locally! In collab with SGLang, you'll learn to: • Deploy fine-tuned LLMs for large scale production • Serve GGUFs for fast inference locally • Benchmark inference speed • Use on the fly FP8 for 1.6x inference ⭐ Guide: https://lnkd.in/g-zYjfjm

474

17 Comments

📣 PSA for those that live under a rock: Apache Airflow 3 is now available! Vikram Koka and Oliver Dykstra from Astro customer Texas Rangers Baseball Club spoke with VentureBeat about everything new in Airflow 3, and why Airflow is the industry-standard for data orchestration and unified DataOps. Read the full article: https://bit.ly/42JtoQC #ApacheAirflow3 #Airflow3 #DataOrchestration #DataOps #DataEngineering

22

1 Comment

Learn how to build an image search app using a series of guided code commits with Developer Advocate Guy Royse. He’ll guide you through generating image embeddings using freely available Hugging Face models and storing them in Redis for fast, scalable semantic search.  The workshop covers: ▶️ What embeddings are and how they work ▶️ How to generate image embeddings and store them in Redis ▶️ Building a semantic search app with Redis, Python, and JavaScript Get access here: https://lnkd.in/gbZ3imbz

44

AI Tech Stack Funding this week: $7.7bn 14 companies that raised this week in the AI infrastructure / tool stack space: Databricks – Venture – $5bn: A data and AI platform that unifies data engineering, analytics, and machine learning on a lakehouse architecture. SambaNova – Series E – $350m: An AI hardware and software company providing infrastructure for AI and machine learning workloads. Fundamental – Series A – $225m: An AI research company focused on overcoming the foundational challenges of machine learning. Goodfire – Series B – $150m: An AI research lab using interpretability to make AI systems understandable, debuggable, and shapeable like software. Midas – Venture – $10m: An AI infrastructure company verifying the mathematical correctness of models through training data validation. ElevenLabs – Series D – $500m: Provides AI tools for speech synthesis, voice cloning, dubbing, and audio generation. Cerebras – Series H – $1bn: An AI computing systems company delivering deep learning hardware, supercomputing, and cloud services. SynthBee, Inc. – Private Equity – $80m: Building a safe, scalable, and reliable computing intelligence platform. adaption – Seed – $50m: Developing AI systems that adapt to changing real-world circumstances. Velatir – Pre-Seed – $1.6m: Delivers AI-native security and compliance solutions enabling safe enterprise adoption of intelligent agents. CNAPS.AI – Seed – Undisclosed: A no-code AI workflow automation platform connecting AI models for visual content generation and intelligence mapping. Gruve – Series A – $50m: Focused on turning enterprise AI strategies into measurable, tangible outcomes. AIRRIVED – Seed – $6.1m: An enterprise-grade agentic AI platform tailored for cybersecurity applications. Berget AI – Pre-Seed – $2.5m: An AI platform focused on inference and agentic infrastructure, enabling businesses to deploy and scale open-source LLMs. #AI #ArtificialIntelligence #AITechStack #AIFunding #AIEcosystem #AIInfrastructure #MachineLearning #ML #DeepLearning #Startups #TechFunding #VentureCapital

14

1 Comment

How well do multimodal LLMs really perform at extracting key insights from documents? From open-source models like Llava-NeXT to API models like GPT-4o and Gemini 1.5 Pro, Quantiphi's Research and Development team put them to the test using real-world datasets across three unique approaches to Key Information Extraction (KIE). Which worked best? Which models stood out? Explore the findings in our latest blog that examines the capabilities of multimodal LLMs in Key Information Extraction (KIE): https://lnkd.in/d8eDHtSu #Quantiphi #ResearchAndDevelopment #SolvingWhatMatters #MultimodalLLMs #AIInsights #AIInnovation

58