Jay Tabaniag

Chains worked perfectly for my product analyst agent when the workflow was simple and tool calling was linear: pull data from Sheets → generate charts with Python → return output. But the moment any conditional logic got added at runtime like “if sample size is low, switch to weekly aggregation” or if it couldn't find the data on it's first try, the flow failed. This is when a more structured framework becomes necessary, a framework that doesn't let the agent get thrown away by logic inside prompts that the tools weren't explicitly designed for, so it can reason, retry and dynamically reroute itself. This week’s breakdown: when simple chains are enough, and when something like LangGraph starts to make more sense. Next week: Building an MCP server hands on. Packaging the tools I designed for the product analyst agent into a locally run MCP server. #TuninginTuesdays #AgenticAI #LangGraph

12

Evaluation and Benchmarks - How to Measure Agent Performance You built an agent. How do you know it actually works? Measuring agent performance is fundamentally harder than measuring model performance. Agents are multi-step systems that use tools, interact with environments, and produce real-world effects. A wrong step early cascades through everything that follows. Traditional accuracy metrics do not capture this complexity. The current benchmark landscape: - SWE-bench: resolve real GitHub issues (code agents) - WebArena: complete browser-based tasks (web agents) - GAIA: multi-step reasoning across tools and information sources - OSWorld: operating system-level task completion Four metrics that matter in practice: - Task success rate: binary completion (did it finish the job?) - Step efficiency: how many steps vs the optimal path - Tool accuracy: right tool, right parameters, right timing - Cost per task: tokens + API calls + compute time The evaluation gap most teams miss: benchmark performance does not equal production performance. An agent scoring 80% on SWE-bench may fail on your specific codebase patterns that benchmarks never tested. The solution is layered evaluation: - Component evals find individual bugs (tool selection, planning, memory) - End-to-end evals find integration failures - Offline evals enable fast iteration during development - Online evals monitor real production performance Building custom evals: define 50-100 representative tasks, create scoring rubrics, use automated graders (a second LLM evaluating the first), and track metrics over every deployment. How are you evaluating your agents beyond demos? Follow for daily insights - where blockchain meets AI, one satisfying swipe at a time. #AgenticAI #AIEvaluation #Benchmarks #SWEbench #AIEngineering

1

If there's anything I've learned recently in the FI space, it's that most banking AI advice is based on opinions. Stoked to see the team Posh do something better. They analyzed millions of AI-powered conversations across 125+ banks and credit unions to see what actually separates high-performing institutions from the rest. The result is a data-driven research of what's actually working with AI in banking. This isn't a list of predictions. It's real production data showing how and why some institutions are consistently outperforming their peers on resolution rates, CSAT, and operational efficiency. What stands out to me most is that the winning institutions aren't spending years debating use cases. They start with a business problem worth solving. Deploy against it. Prove the value. Then scale what works. A crawl-walk-run approach consistently outperforms long evaluation cycles - because the learning only happens once you're live. If your institution is exploring AI or investing in AI and not yet seeing measurable operational impact, the answers are likely in here. Download the full report 👉 https://lnkd.in/g96Tr-4z

14