In this episode of Leaders of Code, Stack Overflow CEO Prashanth Chandrasekar chats with Ramprasad Rai, VP of Platform Engineering at JPMorgan Chase & Co., about the unique challenges of implementing AI in an enterprise environment. They discuss how organizations can balance AI-driven productivity with strict compliance and security requirements by leveraging a community-driven knowledge system that grounds probabilistic AI tools in internal, trusted expertise.
The discussion also:
- Explores why AI models often hallucinate in enterprise environments due to a lack of internal context.
- Highlights how Stack Overflow’s structured Q&A data provides ideal fine-tuning material for the next generation of AI models.
Notes:
- Connect with Ramprasad Rai on LinkedIn.
TRANSCRIPT
Eira May:
Hello everyone, and welcome to the Stack Overflow Podcast. Today, we have another episode of Leaders of Code, where we chat with tech leaders about the work they're doing, the challenges they face, how they build great teams, and much more. My name is Eira May, I'm the B2B editor here at Stack Overflow, and today I'm joined by Stack Overflow CEO Prashanth Chandrasekar, as well as Ram Rai, who is the VP of platform engineering at JPMorgan Chase. How are you guys doing today?
Prashanth Chandrasekar:
I'm doing great. Welcome, Ram, to the podcast.
Ram Rai:
Thank you for having me.
Eira May:
Yeah, so glad you could join us. Prashanth, I know you have a lot of topics that you've been excited to get into with Ram, so I'm just going to step aside and let you have at it.
Prashanth Chandrasekar:
Awesome, thank you, Eira. Ram, welcome again, it's wonderful to see you again after our connection in San Francisco when we were there just a couple of months ago. Thank you for being a Stack Overflow Enterprise customer and supporter, just generally speaking with us. You've got a fascinating background, Ram, I think you've had a range of different roles, Facebook, Meta now and then you've got, obviously, with the JPMorgan AI focus just an amazing career. And so, would love to just maybe get your full introduction, and then we can go from there.
Ram Rai:
Right. So I lead platform architecture and developer experience for JPMorgan digital banking. Before that, I was with Meta, where I optimized their CI/CD systems. Previously, at SAP, I worked on application modeling, rapid development and so forth. That SAP, work in a way, connects to what we do today, abstracting application concepts, express workflow succinctly, just like the way we want to do with prompts. But these days, my focus centers on three things, regulated environments, reliability and LLM applications. I should note that I'm speaking from my personal perspective, not representing JPMorgan, but I look forward to the conversation.
Prashanth Chandrasekar:
Awesome, awesome. Well, fantastic. No, that's just such a great career and you've seen multiple cycles of technology cycles, and you almost alluded to the fact that you were the OG of prompting, in some ways, when you started out several years ago. So I think you've seen these cycles build on top of each other, so that's great. So maybe just to start off, Ram, we obviously at Stack Overflow obsess about community-driven knowledge. Our mission is to be the most vital source for technologists, that's very important for us, our vision rather. And so, for us, I'm curious about your own view on community-driven knowledge and the unique value that it potentially provides to AI tools from your standpoint.
Ram Rai:
So we live in a highly regulated setup with heavy compliance and safety requirements, yet we cannot give up on AI productivity. So we have to be surgical about it, in a way. So if we are talking about code for critical compliance artifacts, like CI/CD pipelines, connectivity to other secure internal systems, we have to be careful. We use deterministic tooling and templates to produce them. We can't entirely trust probabilistic AI with critical infrastructure. So even when AI assists, the critical artifacts are secured, wrapped in guardrails, APIs and so forth.
But the problem is templates don't scale, compliance requirements change quarterly, deployment targets shift, and the engineering maintenance becomes problematic. So for the rest of the code base, we still want to use AI, but responsibly, by grounding AI in our internal reality using solid community knowledge system like Stack Overflow. Think of what AI does without this grounding, it's inherently probabilistic. Think authentication and entitlements, these are areas we cannot go wrong. So the trusted knowledge system changes the equation for us, it's a living documentation, augments AI with context and better solutions, and creates this trust gradient. So deterministic for critical components, community-vetted solutions for tricky coding issues, and AI for speed. So the success, I think, in my opinion, depends on working with these different levels of trust and applying the tools correctly for these situations.
Prashanth Chandrasekar:
Excellent, excellent. That makes a tremendous amount of sense, especially given the context of CI/CD and the context of grounding those tools in the knowledge in the company, that makes a lot of sense. And given our own experience, financial services is easily the largest vertical that we serve, all the big banks are customers of ours, and they all have the same balance of having to think about the regulatory aspects of the industry, that's ultimately what they have to do as part of their job, and ultimately also to optimize for speed and to be able to get stuff done really quickly and to give access to these amazing tools to the community. So a good combination, as you're describing, is grounding in the human fabric of expertise and their knowledge and having that trust layer, and then, of course, leveraging those AI tools off of that foundation so that you're getting the best of both worlds and you can rely on what you're getting out the tools. So that makes a lot of sense.
So speaking of trust then, I think that's a good segue, so you're obviously VP of platform engineering, how do you approach the question of trust in AI-generated code? Because there's a tremendous amount of code being generated now, every developer is going to have access to multiple agents, and the proliferation of code is going to be quite astounding. So how do you in your position manage that, Ram?
Ram Rai:
Right. I think many developers distrust AI accuracy, that's the current reality, and there is a struggle with adoption of AI in a way. So the first question is, why does AI hallucinate? Because it lacks the right context, especially your internal context. AI doesn't know your IDP configuration, token lifetimes, your authentication patterns or your load balance settings, so the training data is thin on this proprietary knowledge. It's never seen your infrastructure, it guesses, and those guesses look convincing sometimes.
So obviously, the solution is to fill the void with the right knowledge. So AI could query organizational knowledge first, what breaks this pattern in our environment, questions like that, and Stack Overflow or systems like that can totally do that, they contain peer-validated solutions. Here is what changes when you do that. The reviewers see the full context, community discussions, votes and so on, and every suggestion comes up with an evidence or a receipt, not just code, but the reasoning behind it and the discussions to support it. And to me, two things stand out. First, the verified context beats model size. Even small models with precise retrieval can outperform large ones. The second is I think that transparency is the new accuracy, showing the white code that you produced produces more trust than a perfect syntax. As you know, there is this code review process in every company, and it's an important part of software development, and having trust over these references will allow the right code to go to production.
Prashanth Chandrasekar:
Yes, absolutely. And ultimately, like you're saying, the context that you have about your organization, specifically the technical context, you used load balancers as an example, that's an excellent example, that's also specific. So you need an ability to store those things in a way that is accessible, can be accessed the right way, it's always up-to-date, always fresh, so that ultimately, these AI agents and tools are working off of that current view of how the company is operating, so that makes a lot of sense. Maybe you can talk a little bit about how do you successfully integrate this context that's, let's say, community knowledge, it's from the experts of the company on a platform like ours, into your workflow inside the company? How does it all flow from an end-to-end perspective?
Ram Rai:
Right. So that's a great question. In a regulated industry, we have unique challenges. There is institutional knowledge, the stuff only your team knows, and we have to adhere to strict compliance practices on top of that. Here is where the traditional approaches fail, the Confluence pages go stale, wikis disappear, and the documentation is always behind. So you have a couple of options, use documentation, but it gets outdated, or you could use a community knowledge platform like Stack Overflow. That's a living documentation and it self-updates and sorts out some of those issues that I mentioned. And on top of that, specifically, Stack Overflow not only captures your enterprise patterns, but also adds collective wisdom from the outside. So this is not generic advice, it's targeted guidance and real answers to the real problem.
There are two things I want to highlight to make this all work, in my opinion. One is context-aware search. The retrieval must understand your programming language, your tech stack, not just keywords, I think this is important to get this to work. And the second is recency. We have to surface the latest security guidelines, not the ones from last year. And the second part to keep the self-sustaining is to have an active learning loop, ensure active contribution. That should be active question-answer discussions with some kind of AI assistance, maybe an MCP server that talks to us like you mentioned. And this incrementally builds institutional expertise, latest performance advice, advice on Java threats, whatever else it might be. I think this makes the knowledge management dynamic, yet reliable.
Prashanth Chandrasekar:
You've given us so many different suggestions along the way, such valuable suggestions as a customer, and you brought up MCP as an example to really fulfill this knowledge intelligence layer vision of our product. About nine months ago or so, we noticed, Ram, that many of our customers were using our APIs, Stack Overflow Enterprise APIs, really, really heavily, and when we went to talk to them, they were all using it for AI use cases, AI search appliances, AI obviously agents, AI coding agents, et cetera, or coding tools. And when we talked to customers like you, it became obvious that we need to go build an MCP server. And then, our team very quickly went and built that, and we went into GA a couple of weeks ago, and just had a tremendous response from customers, because now they're able to really plug in all their AI applications straight into Stack Overflow's enterprise instance, and to be able to get that highly accurate, dynamic, always up-to-date information as part of their workflow, to the question that I was asking.
And by the way, the MCP server is bidirectional, so people are able to, let's say, stay in Cursor or GitHub Copilot and ask the agent to do X, Y, Z on that tool, and then it's able to very intelligently pull from the context, from Stack Overflow directly, from Cursor and GitHub Copilot, and also right back to Stack Overflow straight from Cursor and GitHub Copilot. So the context is always up-to-date, to your point, it's dynamic and always human-curated. So it's been really a great revelation and a great way to engage our customers in a value-added way, so wonderful. So maybe that's a good segue to the impact of all this. When you think about the impact of developer velocity, and just more broadly, business outcomes, JPMorgan's one of the most successful companies in the world, what would you say, Ram, around this entire topic?
Ram Rai:
I want to lay out the obvious benefits, there are lots of benefits. There are two ideas I just was thinking about before this podcast, and something that was important to us even in other contexts as well. First is the understanding, the value of expertise. One expert contributes, and the entire team benefits, part of this knowledge multiplier effect that we are looking for. AI retrieval fans out knowledge instantly. Junior developers can self-serve validated solutions. So in a way, everyone works like an expert, that's what we're looking for, and that's what we get if you follow an approach like this correctly.
And second is appreciating the cost of late-stage bugs and errors. So we use heavyweight penetration testing, security audits, compliance reviews, and a lot of other things. Finding issues late is expensive, it involves multiple teams, delayed releases, and a lot of other issues. So the real value is in catching these errors early, ideally at the coding time, especially if those issues are already known, because different people could make the same error. So I think it's good to catch these issues before they cascade through the system of testing and fixing. And so, what I really want to think about is how to scale expertise without dropping velocity and how to catch known issues early, and a good knowledge platform helps here.
Prashanth Chandrasekar:
Makes sense, that makes sense in the context of the broader outcome, ultimately, because that's what we're all trying to do, right, improve the experience of the developer and their productivity, ultimately. That's ultimately what we're trying to accomplish here, so that makes sense, which then should result in great things for companies like JPMorgan. So maybe turning a little bit towards Stack, you've been such a great partner of ours, looking ahead, from your perspective, given everything that's happening in the industry, very rapidly moving, and everything you know about Stack, both our public platform, obviously our enterprise product, Stack Overflow Enterprise, what role do you think we as Stack Overflow can play, or will play, as AI continues to need this trusted data, from your perspective?
Ram Rai:
I have followed Stack Overflow for years, it's been the go-to place for solving difficult coding questions and software challenges. But if we are talking about today, so in this galaxy of coding tools, you have Claude, Cursor, Copilot and so on, and I feel Stack Overflow holds a unique position, it can empower these other players or work with them. And I see there are two distinct areas where it's going to contribute to the effectiveness of AI coding assistants, or coding in general. First, it's evolving from a website to an intelligent knowledge layer via APIs, MCP or what have you. And the second, it provides perfect training material for AI coding models and helps tune them.
I want to focus here because it's quite insightful. Just to give you a bit of a context, Raw LLMs have the potential, but they lack direction, they're not useful right away. Fine-tuning is what gives them the direction and makes them useful. You've heard of, let's say, instruction models, which are trying to answer questions directly, or you have reasoning models, like o1, which think through the problem before responding, you have a question, reasoning and then an answer, and Stack Overflow's QA structure maps perfectly to both of these approaches.
I think there are three fine-tuning methods, at least where I could think, that Stack Overflow data can help directly. First is instruction tuning of these models. The QA paths are already formatted correctly, how do I do... and here is the solution. You have a data set of questions and answers, and this is perfect for instruction models. And second, there is so-called preferential learning. Occasionally, when we use ChatGPT or Claude, it pops up two options, two answers, and you have to pick one, and they're collecting preference signals. And Stack Overflow naturally has comparative signals, an answer with 400 upwards is obviously better than the answer with two upwards, that's a second signal there. And the third one is for this process level reasoning, for reasoning models. You teach not just the answers, but also how to think. So the discussions naturally break down problem solving in Stack Overflow, you have option A, it led me here, option B, I got this exception, and finally, you have some other approach. So this is real reasoning data, not synthetic reasoning trace and we can teach LLMs how to think like good programmers.
Looking forward, just to summarize this, I think Stack Overflow will become a trusted validation layer where AI goes to verify. And second, I think it serves training data that can make AI coding assistants effective, and even make small language models as good as the large ones, or maybe even better sometimes.
Prashanth Chandrasekar:
Indeed, specialized models, yeah, that makes a lot of sense. I think that you're absolutely right, Ram, in that over the past couple of years, we've been very fortunate to partner with all the big AI labs and all the big cloud companies that are in the AI LLM space. So everybody from the OpenAIs and the Google Geminis to everybody else in the ecosystem, they're all partners of ours, because they recognize effectively exactly what you said, is that the depth of knowledge that we have from the experts on our platform, which are 100 million people over the past 16, 17 years of accumulating that knowledge, and we have a phenomenal number of new questions being asked on the platform now through our AI assist, which is different from our traditional Q&A mode. We have about 60 billion tokens plus on the platform in aggregate, so that's a lot of data.
But to your point, the reasoning aspects, it's almost like a brain, because people asked the question, they debated it, there was the correct answer that got voted to the top, but there is a very significant number of comments and discussions happening between users on why a particular answer is good. So it is literally like a human brain, or very much like it, because it's representing the collective reasoning of a lot of experts, and that continues, and now we're seeing... So all of that, just to finish that thought, is that a lot of the AI labs obviously find that to be valuable, because we're literally the only place on the internet that has that level of depth of knowledge for technology-related subjects.
And now, with our AI assist feature, which is on stackoverflow.com, you can go to stackoverflow.ai, there's a tremendous number of AI-first conversations happening on the site, where people are using a ChatGPT-like format, where they're engaging with our AI which is grounded in the knowledge on the site, and that in itself is creating all sorts of exchanges between users. To your point around the validation layer, they're able to import conversations they're having, let's say, in Claude about coding, but they're stuck based on, let's say, specific issue, and then our AI then sorts that out. So then, we've also now built an MCP server for the public data set, so people are also able to plug into that in the context of their coding tools. So just a fascinating number of new things. But you're right in that our role has evolved, and not only will we be the community to connect people, but we will power learning and unlock growth for developers and technologists as they engage with the new tools of the ecosystem.
Fantastic. Well, thank you, Ram, that's wonderful insights from you. And I'd love to maybe make sure that the audience knows how to reach you, so how can they reach you?
Ram Rai:
I'm on LinkedIn. If you have questions on what we already talked today or something else, please reach me on LinkedIn.
Prashanth Chandrasekar:
Wonderful. And same thing for me, I'm on LinkedIn or X, and you can ping me directly on that. So fantastic, Ram, thank you again. And Eira, back to you.
Eira May:
Thank you for listening to this week's episode of Leaders of Code. If you have suggestions for topics you'd like us to cover or guests you want to hear from on future episodes, you can email podcast@stackoverflow.com. My name is Eira May, and I am the B2B editor at Stack Overflow. Thanks, and see you on the next episode of Leaders of Code.