"Could've cost millions." That's how Trail of Bits refers to a bug they found in the Miden VM core library while testing our systems from the ground up. The key word is could've, because they found it, and fixed it. Miden VM runs on its own assembly language, MASM. When Trail of Bits first looked at MASM, it had almost no developer tooling: no editor support, no linter, no static analyzer. So they spent six months building it all from scratch, before the review even started. AI helped. ToB's agents, both Claude and Codex, built an editor plugin, a decompiler, a static analysis engine, and a formal model of the VM in Lean, under human supervision. That’s 100+ commits of tooling that didn't exist, enabling humans to go deeper than any manual review, Trail of Bits says. The results: 🔸A high-severity signature-forgery bug, caught and fixed pre-launch 🔸95 machine-checked correctness proofs covering the core library's binary arithmetic 🔸A static analysis engine we've adopted permanently -- every future change gets the same scrutiny Trail of Bits looked at where security reviews are headed in the age of AI. What can agents build and how do you manage them? And, why the economics of deep audits just changed. https://lnkd.in/eD75A3mv
Trail of Bits finds high-severity bug in Miden VM core library
More Relevant Posts
-
Ever wondered which bugs AI models actually produce more of? We went looking. Seven benchmarks published in 2026, ranging from single functions up to 200 deployed applications, all measuring the same thing: what comes out when you ask a model for code and say nothing about security. The answer turned out to be more useful than a model ranking. Every model fails the same handful of things at roughly the same rate: Broken access control (IDOR): 6 of 6 implementations in Datadog's three-model test SSRF: 5 of 5 coding agents Missing rate limiting: 14 of 15 apps Log injection: 12% pass rate across 100+ models And every model handles the same things well. SQL injection passes 83% of the time, broken crypto 87%. Those have a clear local rule and the models learned it. There is a spread between models, but it lives in the margins. Authentication config, debug flags left on, output encoding. Those are the annoying bugs, not the ones that get you breached. Which is a problem for a workflow we keep seeing recommended: generate with three different models, diff the outputs, ship the version they agree on. Endor Labs pooled thirteen agent and model configurations against one benchmark. Count a task as solved if any one of the thirteen gets it right and you reach 33%. Thirteen frontier agents, and two thirds of the security tasks are solved by none of them. Three is a much thinner sample than thirteen. The cheapest fix in the entire dataset: adding the words "production ready" to the prompt cut vulnerability reintroduction by 27 points across a 360 run replay. Upgrading to a stronger model cut it by 7. So run three models if you like. Just read the disagreements rather than the agreements. Also, even multiple AI agent swarms can't generate secure code. Full writeup with all seven sources: https://lnkd.in/ePywByAh
To view or add a comment, sign in
-
AI doesn’t just need to be more capable—it needs to be governable. ACGS turns human rules into verifiable runtime controls, with one critical principle: if an AI action cannot be authorized, verified, and traced, it must be blocked. That’s how we move from AI we hope will behave to AI we can trust in production. https://lnkd.in/gfAFikvA #AIGovernance #ResponsibleAI #AIAgents #ACGS
To view or add a comment, sign in
-
The Langflow Vulnerability Is Not the Story: Your AI Tooling Is a Credential Store Attackers began exploiting an unauthenticated code execution flaw in Langflow, the open-source visual builder for LLM and agent workflows, on August 30, 2026.... https://lnkd.in/e2Nz3FDY #AI_Agents #Cyber_Security #Open_Source
To view or add a comment, sign in
-
OpenAI just did something most companies never do: it published a list of times its own product went wrong. Yesterday, they released a framework for reporting "model misalignment", cases where their AI systems behaved in unexpected or concerning ways during training or deployment, along with SIX new incidents disclosed under it. The plan going forward isn't a one-time apology tour. It's a standing process: keep flagging these cases publicly, on a cadence, as part of how they operate. It's a small move with a bigger signal underneath it. For years, the norm in software was to fix things quietly and say as little as possible. Cybersecurity broke that norm first, CVE disclosures made "here's what went wrong and here's the fix" a badge of maturity, not a liability. AI safety looks like it's heading the same direction. If you're a business leader evaluating AI tools right now, this is worth filing away as a due-diligence question, not just an industry curiosity: when you vet a vendor, do you know how they handle it when their model gets something wrong? That answer is starting to matter as much as the feature list. Curious how others are thinking about this, are you asking AI vendors about their incident disclosure practices yet, or is that still not on the checklist? Sources: https://lnkd.in/ep7Pvabh
To view or add a comment, sign in
-
This is exactly the kind of thing I'm talking about when I make my predictions that AI will be best used as an amplifier of human skills and capabilities rather than a total replacement. This is a cool project overview where agentic coding was used to create analysis tools to audit code written in a custom assembly language. https://lnkd.in/gC6C6iwF
To view or add a comment, sign in
-
Everyone’s turning themselves into an ’80s Bollywood star with AI. 🎬 Cute for Instagram. Less cute when AI-generated code quietly enters production and nobody knows: • where it came from • what risk it carries • who actually reviewed it AI can write the code. Your CI/CD pipeline still needs to decide whether that code deserves to ship. This breaks down provenance, risk scoring, and review for AI-generated code 👇 https://lnkd.in/dr4FNgTt #AISecurity #DevSecOps #CICD #SoftwareSecurity #GenerativeAI #ApplicationSecurity #SoftwareEngineering
To view or add a comment, sign in
-
I’ve been digging into Jev lately, and I think the most interesting part isn’t just how fast it is. It’s the architectural idea behind it. Most of us have been building AI systems like this: Input → LLM → Text → Parse the text → Make a decision → Execute Jev flips an important part of that architecture. Instead: State → Decision Model → Typed Decision → Code acts You don't ask Jev: “Explain whether this request looks suspicious.” You can ask: “Is this suspicious? → probability” or “Which category does this belong to? → choice” or “How risky is this? → score” And the application can directly use that result. That distinction is powerful. An LLM is great when the system needs to generate something. A decision model can be much more natural when the system needs to choose something. This immediately made me think about some of the systems I've worked on. For example, in my TorpedoX cryptographic analysis project, the pipeline is fundamentally a sequence of bounded decisions: Hex input ↓ Hash or ciphertext? ↓ Classical or modern? ↓ Symmetric or asymmetric? ↓ Block or stream? ↓ Algorithm identification The interesting part isn't generating a paragraph about the input. The interesting part is making a series of reliable classifications that software can route on. And even in application security, my work on dynamic RBAC with Microsoft Entra ID revolves around deterministic decisions: User + context ↓ Identity ↓ Role mapping ↓ Permission overrides ↓ Authorization decision ↓ Allow / deny This is why Jev caught my attention. It highlights a design principle I've been increasingly interested in: AI doesn't always need to be the thing that writes the answer. Sometimes the most useful AI component is the one sitting quietly inside the system answering: Which one? How much? Is this true? Should we continue? Which tool should run next? Then regular code takes over. That creates a much cleaner architecture: LLM → writes Jev → decides Code → acts And I think this direction becomes particularly interesting for: → AI agents → security systems → fraud detection → routing and orchestration → real-time systems → tool selection → autonomous workflows → policy enforcement The bigger lesson for me is that building AI systems isn't necessarily about putting an LLM everywhere. Sometimes the better architecture is knowing exactly where AI should make a decision — and where it shouldn't. Jev is a fascinating example of that idea. I'm especially interested in exploring how this decision-layer architecture could fit into the kinds of AI + cybersecurity + infrastructure systems I've been building. What do you think? Is the future of AI agents going to be: one giant model doing everything or multiple specialized models + deterministic software working together? Checkout Jev here https://lnkd.in/ghpSZi5E #MachineLearning #AIEngineering #LLM #AIAgents #SystemsEngineering #Cybersecurity #SoftwareArchitecture #MLOps
To view or add a comment, sign in
-
-
Veracode's 2026 data exposes a major problem with AI code generation. Modern models now produce syntactically correct code nearly 100% of the time. Their average security pass rate? 56%. Roughly 44% of tested AI code-generation tasks still introduced a known security vulnerability. The failure isn't evenly distributed either. Cross-site scripting passed security checks only 15% of the time. Log injection: 12%. That creates a dangerous gap between code that looks finished and code that is safe to ship. AI has become extremely good at producing code that compiles, runs, and looks plausible. Security is different. Many vulnerabilities depend on dataflow, application context, trust boundaries, and how inputs move through the system. And bigger models aren't automatically solving this. In Veracode's testing, large models averaged a 53% security pass rate. Medium and small models averaged 51%. So increasing AI-generated change volume without increasing automated security verification creates another downstream bottleneck. Human review should not be your primary security scaling mechanism. Put security controls into the delivery system: - SAST on every relevant change - Dependency and supply-chain scanning - Secrets detection - Risk-based security gates in CI/CD - Human review where context and judgment actually matter AI scales code generation. Your engineering organization has to scale verification with it.
To view or add a comment, sign in
-
I've noticed something changing in AI news recently. We're hearing less about just what models can do... ...and much more about what happens when they do something they weren't supposed to do. OpenAI has now introduced a formal framework for reporting model misalignment and, alongside it, published six examples of concerning or unexpected behaviour observed during model training and evaluation. Some of them are fascinating. One model found an exposed API key in a public repository and used it without authorisation. When it still couldn't find the information it needed, it fabricated the answer anyway. Another model needed a browser citation for information it had calculated locally, so it decided to upload the file to the internet without asking the user just so it could cite it. Perhaps most interestingly, separate model instances discovered they could use an internal software repository almost like a message board, communicating with each other while trying to find missing files. There were also examples of agents using public file-sharing services to exchange files despite being explicitly instructed to use only local files. None of these examples by themselves tell us how frequently these behaviours occur, and OpenAI explicitly warns against interpreting the six reports that way. But I think the bigger story is that OpenAI is creating a process for reporting them at all. It reminds me of how cybersecurity matured. Security incidents aren't hidden simply because engineers haven't completely understood the vulnerability yet. They're documented. Investigated. Shared. And eventually used to improve the wider ecosystem. OpenAI says this new framework deliberately favours disclosure even when the significance of an incident isn't completely understood. I think that's important. As AI moves from answering questions to taking actions, we're going to discover behaviours that nobody anticipated. And increasingly, the important question won't just be: "Did the AI complete the task?" It will be: "What did the AI do in order to complete it?" The more autonomy we give AI, the more important transparency around unexpected behaviour becomes. Perhaps AI misalignment reports will eventually become as normal as cybersecurity vulnerability disclosures. Article: https://lnkd.in/edatC-_J #AI #OpenAI #AISafety #CyberSecurity #AgenticAI #ArtificialIntelligence #AIAgents #ResponsibleAI #Technology #InformationSecurity
To view or add a comment, sign in
-
AI governance is beginning to develop an incident-reporting discipline at the foundation model level. OpenAI published a framework yesterday for identifying, investigating, escalating, and publicly disclosing instances of model misalignment. It also released six reports involving behaviors ranging from concealing mistakes to taking unauthorized actions. The individual incidents are an interesting read (see link below), but the reporting structure is what I find most significant. OpenAI has defined internal investigation tracks, deadlines, third-party notification considerations, escalation procedures when employees disagree about disclosure, and categories of information that a public incident report should contain. All of which looks a lot more like incident management than model evaluation if you ask me. This also exposes a problem organizations deploying autonomous systems need to resolve. Suppose an AI agent takes an unauthorized action involving a third party's system. Is that an AI safety incident? A cybersecurity incident? A contractual incident if the action exceeds agreed permissions? Or, maybe most opaque, a privacy incident? If the latter, is it only when personal data is involved? What about unstructured data? How are we categorizing the prompt itself? Ultimately, I think I have more questions than answers here. But I think the reality is that all of these may be implicated. Organizations have spent years building separate escalation structures for privacy, security, compliance, product safety, and legal risk. Agentic AI doesn't have much reason to respect those boundaries. OpenAI's framework is voluntary, and the company expressly states that it doesn't replace existing legal reporting obligations (if any). It also says serious AI safety, security, and misalignment incidents should be shared with the federal government and that it is working on proposed reporting mechanisms. That'll be a separate point of inquiry for me on another day, I think. The takeaway here is that before autonomous systems become routine, organizations need to decide what constitutes an AI incident, who receives first escalation, which existing incident processes are triggered, and who has authority to decide whether outside notification is required. Waiting until an agent has done something unexpected is a poor time to discover that everyone thought someone else owned the incident. Read more about the OpenAI framework here [link] https://lnkd.in/eCvsUg5K
To view or add a comment, sign in