Director of Engineering focused on data platforms, streaming infrastructure, and AI-native engineering systems.
Over the past 14+ years I’ve worked on large-scale distributed platforms supporting enterprise engineering organizations. Most recently I led the Spark Streaming and Messaging platform organization at Walmart, operating infrastructure used by thousands of production pipelines and thousands of engineers across the company.
My work sits at the intersection of:
• streaming data platforms
• distributed systems infrastructure
• lakehouse data architectures
• AI-native development workflows
I’m particularly interested in how modern engineering organizations can build more capable systems with smaller teams by combining strong platform architecture with AI-assisted engineering workflows.
LinkedIn
https://www.linkedin.com/in/brandon-benge-3b57a547/
At Walmart I owned the enterprise Spark Streaming platform, responsible for:
• supporting 3,500+ production data pipelines
• operating ~70 Kubernetes clusters
• serving 3,000+ engineers and data practitioners
• managing ~$26M in annual infrastructure spend
My role combined platform architecture, operational stewardship, financial accountability, and engineering leadership.
Key areas of focus included:
• stabilizing large-scale distributed systems under operational pressure
• improving platform reliability while reducing infrastructure cost
• introducing GitOps-based platform delivery models
• rebuilding teams and restoring sustainable execution during organizational change
More recently, I’ve been exploring how AI-native development workflows can reshape how platform teams design, build, and operate complex systems.
I enjoy writing about how modern platforms evolve to support AI systems and real-time decision making.
From Data Pipelines to Real-Time Decisions
Streaming lakehouse architecture for real-time ML platforms
https://medium.com/@bengebc/from-data-pipelines-to-real-time-decisions-9b53dba8fc9d
Owning Your AI
Lessons learned building an open agentic AI system
https://medium.com/@bengebc/owning-your-ai-what-i-learned-building-apache-agentic-2f49f654c77b
Outside of my professional work I build reference architectures and experimental platforms to explore emerging infrastructure patterns.
These projects are intended as executable architecture examples, demonstrating how modern systems can be assembled end-to-end.
Example Data Pipeline with ML
https://github.com/brandon-benge/example-data-pipeline-w-ml
Demonstrates how streaming pipelines, lakehouse architecture, and ML inference services can work together to support real-time operational decisions.
Patient event streaming systems:
https://github.com/brandon-benge/patient-events-write-platform
https://github.com/brandon-benge/patient-events-stream-platform
Exploring how event-driven architectures support large-scale operational systems.
Experiments with agent-driven development and AI-assisted engineering workflows.
https://github.com/brandon-benge/apache-agentic
https://github.com/brandon-benge/RAG-Workflow
https://github.com/brandon-benge/langchain_autocommit
Prometheus Metric Summarizer
https://github.com/brandon-benge/prometheus-metric-summarizer
Exploring how AI systems can help engineers interpret large volumes of operational metrics.
I’m currently exploring how engineering organizations can evolve toward:
• AI-native development workflows
• smaller teams building more capable platforms
• streaming-first data architectures
• real-time ML decision systems
• agentic AI tooling integrated into engineering workflows
The long-term goal is simple:
enable engineering teams to do more with less while operating increasingly complex systems.
If you’re working on similar platform challenges, I’m always interested in connecting.