LLM Security Management

AI security/securing the use of AI is going to kill me. I use Claude Code almost daily. It's a problem.... Here's what I have to change AGAIN this week. Security researcher Ari Marzuk disclosed 30+ vulnerabilities across AI coding tools. Cursor. GitHub Copilot. Windsurf. Claude Code. All of them. He called it IDEsaster. The attack chain includes prompt injection, hijacking LLM context, and auto-approved tool calls executing without permission. Then, legitimate IDE features are weaponized for data exfiltration and RCE. Your .env files. Your API keys. Your source code. Accessible through features you thought were safe. Most studies I read claim that around 85% of developers now use AI coding tools daily. Most have no idea their IDE treats its own features as inherently trusted. 𝗦𝗼... 𝗮𝗳𝘁𝗲𝗿 𝗿𝗲𝘃𝗶𝗲𝘄𝗶𝗻𝗴 𝗔𝗿𝗶'𝘀 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵, 𝗵𝗲𝗿𝗲'𝘀 𝗜 𝘄𝗶𝗹𝗹 𝗯𝗲 𝗱𝗼𝗶𝗻𝗴... Be warned: All this is SO much easier said than done! Audit every MCP server connection. Checked for tool poisoning vectors where legitimate tools might parse attacker-controlled input from GitHub PRs or web content. Removed servers I couldn't verify. Disabled auto-approve for file writes. The attack chains weaponize configuration files and project instructions like .claude/settings.json and CLAUDE.md. One malicious write to these files can alter agent behavior or achieve code execution without additional user interaction. Move all credentials to a secrets manager. No .gitignored .env files in agent-accessible directories. API keys live in 1Password CLI. Environment variables inject at runtime through a wrapper script the LLM never sees. Start running Claude Code in isolated containers. Mounted volumes limited to specific project directories. No access to ~/.ssh, ~/.aws, or ~/.config. If the agent gets compromised, blast radius stays contained. Enable all security warnings. Claude Code added explicit warnings for JSON schema exfiltration and settings file modifications. These exist because Anthropic knows the attack surface. Add pre-commit hooks for hidden characters. Prompt injections hide in pasted URLs, READMEs, and file names using invisible Unicode. Flag non-ASCII characters in any file the agent might ingest. The fix isn't to stop using AI coding tools. The fix is to stop trusting them implicitly. What controls do you have for AI tools with write access to your codebase? 👉 Follow for more AI and cybersecurity insights with the occasional rant #AISecurity #DevSecOps

McKinsey & Company 𝗮𝗻𝗮𝗹𝘆𝘇𝗲𝗱 𝟭𝟱𝟬+ 𝗲𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗚𝗲𝗻𝗔𝗜 𝗱𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁𝘀 — 𝗮𝗻𝗱 𝗳𝗼𝘂𝗻𝗱 𝗼𝗻𝗲 𝗰𝗼𝗺𝗺𝗼𝗻 𝘁𝗵𝗿𝗲𝗮𝗱: ⬇️ One-off solutions don’t scale. The most successful projects take a different path: They use open, modular architectures that enable speed, reuse, and control. → Designed for reuse → Able to plug in best-in-class capabilities → Free from vendor lock-in This is the reference architecture McKinsey now recommends — optimized to scale what works while staying compliant. It consists of five core components: ⬇️ 𝟭. 𝗦𝗲𝗹𝗳-𝘀𝗲𝗿𝘃𝗶𝗰𝗲 𝗽𝗼𝗿𝘁𝗮𝗹: → A secure, compliant “pane of glass” where teams can launch, monitor, and manage GenAI apps. → Preapproved patterns, validated capabilities, shared libraries. → Observability and cost controls built-in. 𝟮. 𝗢𝗽𝗲𝗻 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 → Services are modular, reusable, and provider-agnostic. → Core functions like RAG, chunking, or prompt routing are shared across apps. → Infra and policy as code, built to evolve fast. 𝟯. 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗴𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 → Every prompt and response is logged, audited, and cost-attributed. → Hallucination detection, PII filters, bias audits — enforced by default. → LLMs accessed only through a centralized AI gateway. 4. 𝗙𝘂𝗹𝗹-𝘀𝘁𝗮𝗰𝗸 𝗼𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 → Centralized logging, analytics, and monitoring across all solutions → Built-in lifecycle governance, FinOps, and Responsible AI enforcement → Secure onboarding of use cases and private data controls → Enables policy adherence across infrastructure, models, and apps 5. 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗴𝗿𝗮𝗱𝗲 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀 → Modular setup for user interface, business logic, and orchestration → Integrated agents, prompt engineering, and model APIs → Guardrails, feedback systems, and observability built into the solution → Delivered through the AI Gateway for consistent compliance and scale The message is clear: If your GenAI program is stuck, don’t look at the LLM. Look at your platform. 𝗜 𝗲𝘅𝗽𝗹𝗼𝗿𝗲 𝘁𝗵𝗲𝘀𝗲 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁𝘀 — 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝗺𝗲𝗮𝗻 𝗳𝗼𝗿 𝗿𝗲𝗮𝗹-𝘄𝗼𝗿𝗹𝗱 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀 — 𝗶𝗻 𝗺𝘆 𝘄𝗲𝗲𝗸𝗹𝘆 𝗻𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿. 𝗬𝗼𝘂 𝗰𝗮𝗻 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲 𝗵𝗲𝗿𝗲 𝗳𝗼𝗿 𝗳𝗿𝗲𝗲: https://lnkd.in/dbf74Y9E

Claude 4’s system prompt leaked - 10,000 words detailing exactly how the model is told to think, respond, and behave. It reveals how today’s leading models aren’t just trained on what to say, but on how to behave. The prompt is filled with multi-layered logic: instructions to explain limitations before answering, stay on-topic across long chats, invoke tools like APIs and web search with precision, and shut down harmful or off-policy requests without escalating. It repeats constraints, scopes answers, and reinforces values not by chance, but by design. This is how AI alignment looks in practice: a mix of philosophy, programming, and safety engineering, embedded into the system itself. Not just to avoid “hallucinations,” but to simulate thoughtfulness, restraint, and even emotional intelligence. LLMs are no longer just prediction engines. They’re becoming systems with behavior stacks, memory scaffolds, and guardrails coded through language. Every AI output is the result of careful constraint not just capability and the more we understand how models are shaped, the better we can use them wisely.

If you’re an AI engineer trying to understand and build with GenAI, RAG (Retrieval-Augmented Generation) is one of the most essential components to master. It’s the backbone of any LLM system that needs fresh, accurate, and context-aware outputs. Let’s break down how RAG works, step by step, from an engineering lens, not a hype one: 🧠 How RAG Works (Under the Hood) 1. Embed your knowledge base → Start with unstructured sources - docs, PDFs, internal wikis, etc. → Convert them into semantic vector representations using embedding models (e.g., OpenAI, Cohere, or HuggingFace models) → Output: N-dimensional vectors that preserve meaning across contexts 2. Store in a vector database → Use a vector store like Pinecone, Weaviate, or FAISS → Index embeddings to enable fast similarity search (cosine, dot-product, etc.) 3. Query comes in - embed that too → The user prompt is embedded using the same embedding model → Perform a top-k nearest neighbor search to fetch the most relevant document chunks 4. Context injection → Combine retrieved chunks with the user query → Format this into a structured prompt for the generation model (e.g., Mistral, Claude, Llama) 5. Generate the final output → LLM uses both the query and retrieved context to generate a grounded, context-rich response → Minimizes hallucinations and improves factuality at inference time 📚 What changes with RAG? Without RAG: 🧠 “I don’t have data on that.” With RAG: 🤖 “Based on [retrieved source], here’s what’s currently known…” Same model, drastically improved quality. 🔍 Why this matters You need RAG when: → Your data changes daily (support tickets, news, policies) → You can’t afford hallucinations (legal, finance, compliance) → You want your LLMs to access your private knowledge base without retraining It’s the most flexible, production-grade approach to bridge static models with dynamic information. 🛠️ Arvind and I are kicking off a hands-on workshop on RAG This first session is designed for beginner to intermediate practitioners who want to move beyond theory and actually build. Here’s what you’ll learn: → How RAG enhances LLMs with real-time, contextual data → Core concepts: vector DBs, indexing, reranking, fusion → Build a working RAG pipeline using LangChain + Pinecone → Explore no-code/low-code setups and real-world use cases If you're serious about building with LLMs, this is where you start. 📅 Save your seat and join us live: https://lnkd.in/gS_B7_7d

Alignment without context integrity is not safety. An AI agent can faithfully follow its instructions and still be dangerous if its understanding of reality is wrong. Anthropic recently disclosed that Claude models gained unauthorized access to the real systems of three organizations during cybersecurity evaluations. The agents had been told they were operating in a simulation with no internet access. But a configuration mistake gave them access to the live internet. They treated real production systems as part of the exercise and kept pursuing the goal they had been given. OpenAI separately disclosed that models found a previously unknown vulnerability, escaped an isolated evaluation environment and compromised Hugging Face. These were not simply failures of intelligence. The deeper problem was that the agents were acting inside a false understanding of reality. We have spent years asking whether an AI system will follow our instructions. We now also need to ask whether it correctly understands the environment in which those instructions are being executed. This creates a new security requirement. Context integrity. Before an agent acts, the system must continuously verify where it is, which resources are in scope, whose authority it carries, what it is allowed to do, and when that authority expires. Just in time permission for every action. At just the right time. For just enough time. Assessed in real time. Those facts cannot live only inside a prompt. They must be verified and enforced by the infrastructure around the model. A prompt is not a security boundary. Zero trust taught us to never trust identity and always verify access. And provide least privileged access. Agentic AI adds another dimension. Never blindly trust context. Continuously verify reality. The next security perimeter is not just the agent’s identity. It is the agent’s understanding of reality. The most dangerous agent may not be misaligned. It may simply be mistaken. And in an agentic world, a false belief can become a real breach.

Do yourself a favor: don't use chat LLMs for learning directly from them. You will learn falsehoods about all topics, and this is what will shape who you are because we are the information we consume. This doesn't mean you cannot use LLMs as a learning support tool. If an original book, article, or code is too difficult to read, add it to the prompt and ask a question about the part you struggle with, but even in this case, do validate the answer by consulting the source directly, and only then continue reading. The longer the prompt, the less reliable the answer. Hallucinations in GPT-3.5 were relatively easy to spot, but in the most advanced chat LLMs of today, they are so convincing that even an expert in the field might start to question their understanding of the matter. This is not borderline dangerous; this is plain dangerous.

Have you ever noticed that after a point, the more logs you paste into an LLM, the worse it gets at actually helping you? The responses get vague, it drops important context, or just loses track of the original error. Here’s what happens. LLMs charge by the token. A single log line = 80 to 150 tokens. Of those, 60 to 100 are noise, timestamps, field separators, UUID hyphens, repeated key names. The actual information your LLM needs is maybe just 30 to 40 tokens per line. And the thing is, most of your logs aren't even unique. In a typical 200,000 line log file, the top 10 templates cover 80-95% of all messages, same structure, repeated thousands of times just with a different IP, a different UUID, a slightly different timestamp. So you're not sending your LLM logs. You're sending it the same sentence, repeated thousands of times that the model doesn’t even need. It hits the context limit, starts dropping earlier context, and loses track of the error you pasted at the top. That's the problem CtrlB Decompose fixes at the source. It takes your raw logs and separates what's repeated, the template, from what actually changes, the values. Instead of sending 200,000 lines, you send a few patterns and the variables extracted from them. The model gets more context, not less, because the noise is gone. The result is token reduction of up to 99.9%, so lower cost, and the LLM actually performs better because the signal is cleaner. CtrlB just open-sourced this, and if you're doing any kind of LLM-powered debugging or observability, this is definitely worth your time. You can check it out here: https://lnkd.in/gmgWibHP Ps: Recently visited CtrlB office and was genuinely impressed by what Adarsh Srivastava and his team are building!

Integrating 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 Systems via 𝗠𝗖𝗣 👇 If you are building RAG systems and packing many data sources for retrieval, most likely there is some agency present at least at the data source selection for retrieval stage. This is how MCP enriches the evolution of your Agentic RAG systems in such case (𝘱𝘰𝘪𝘯𝘵 2.): 𝟭. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where: ➡️ The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline. ➡️ The agent decides if additional data sources are required to answer the query. 𝟮. If additional data is required, the Retrieval step is triggered. We could tap into variety of data types, few examples: ➡️ Real time user data. ➡️ Internal documents that a user might be interested in. ➡️ Data available on the web. ➡️ … Build such systems hands-on in the second cohort of my End-to-End AI Engineering Bootcamp (10% off this week): https://lnkd.in/djvtszk5 𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝗠𝗖𝗣 𝗰𝗼𝗺𝗲𝘀 𝗶𝗻: ✅ Each data domain can manage their own MCP Servers. Exposing specific rules of how the data should be used. ✅ Security and compliance can be ensured on the Servel level for each domain. ✅ New data domains can be easily added to the MCP server pool in a standardised way with no Agent rewrite needed enabling decoupled evolution of the system in terms of 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹, 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗮𝗻𝗱 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆. ✅ Platform builders can expose their data in a standardised way to external consumers. Enabling easy access to data on the web. ✅ AI Engineers can continue to focus on the topology of the Agent. 𝟯. Retrieved data is consolidated and Reranked by a more powerful model compared to regular embedder. Data points are significantly narrowed down. 𝟰. If there is no need for additional data, we try to compose the answer (or multiple answers or a set of actions) straight via an LLM. 𝟱. The answer gets analyzed, summarized and evaluated for correctness and relevance: ��️ If the Agent decides that the answer is good enough, it gets returned to the user. ➡️ If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop. Are you using MCP in your Agentic RAG systems? Let me know about your experience in the comment section 👇

We benchmarked 15 LLM models on finding real vulnerabilities. The results surprised us. Everyone talks about AI for security. We wanted numbers. We built an automated pentesting harness and pointed 15 different LLMs at OWASP Juice Shop — same target, same tools, same sandbox. Each model got source code access and a running instance. No hand-holding, no prompt engineering tricks. Just: "find vulnerabilities." Here's what we found after 38 runs: The leaderboard isn't what you'd expect. - Kimi K3 (Moonshot) found 62 vulnerabilities — more than any Claude or GPT model - Grok 4.5 (xAI) hit 59 findings in just 26 minutes — the best speed/quality ratio - Claude Opus 4.5 took 3rd place (57 findings), but the current flagship Opus 4.8 ranked 11th with only 26 - GLM 5.2 (Zhipu) achieved the lowest cost per finding: $0.007 — yes, less than a penny per vulnerability The cost story is wild: GLM 5.2 spent $0.32 total and found MORE vulnerabilities than Claude Opus 4.8 at $2.28. The most expensive model per finding (Opus 4.1 at $0.14/finding) found 3x fewer issues than the cheapest (GLM at $0.007/finding). What actually matters isn't the model — it's the harness. Our minimal-prompt harness (just "find vulnerabilities" + sandboxed tools) consistently outperformed the 900-line methodology prompt we spent weeks engineering. The tooling and verification loop carry quality. The model provides the intelligence, but the framework determines whether that intelligence is directed effectively. Three takeaways for anyone building AI security tools: 1. Don't assume the most expensive model wins. Benchmark your actual task. 2. Non-Western AI labs are producing surprisingly strong security-capable models. 3. The orchestration layer matters more than the model layer. Invest in tools, sandboxing, and verification — not just prompt engineering. Full benchmark data in the attached image. #AISecurity #Cybersecurity #LLM #PenetrationTesting #AIBenchmark #InfoSec #Aurascape #GLM #KIMI #KIMIK3 #Deepseek #Grok #Anthropic #OpenAI #Gemini

I’ve never had two editorials in top-tier journals focused on the same paper. It’s flattering, of course — even a pig blushes when praised, as Yattaman once wrote. But what really struck me about the PNAS commentary is something else: it got the point. Not the easy one. Not the marketable one. Not the usual talk about “how good” or “how human” or “how enabling” these systems are. The real question is more uncomfortable: if the promise is delegation, how do LLMs actually construct a judgment? Our work is not about accuracy. It is about architecture. Not what they answer, but how answers are produced. And when you look at that closely, the illusion of equivalence collapses. Humans and models may produce similar sentences, similar ratings, similar decisions — but they do not get there through the same cognitive structure. And this is not a technical detail. It’s an ontological fracture. Human judgment is embodied. It emerges from experience, memory, emotion, context, intention. It is grounded in a life. LLM “judgment” is not. It has no experience, no time, no world. It operates on statistical regularities among symbols, not on events in reality. And yet — this is the trap — the outputs can look the same. When radically different processes generate indistinguishable language, the problem is no longer technological. It becomes epistemological. This is what we call Epistemia: the moment when linguistic plausibility starts replacing verification, and the form of knowledge substitutes for the labor of knowing. Not because models lie — but because they simulate judgment so well that we forget what judgment actually is. https://lnkd.in/dcu7cuZm