Threat Modeling AI Systems: A Practical Guide for LLM and Agentic Applications

Every security team is now being handed AI features to review with a process that was designed for CRUD applications: a form, a service, a database, a queue. Two instincts follow, and both are wrong. The first says this needs a whole new discipline, so the team goes looking for an AI security framework and spends a quarter evaluating one instead of reviewing anything. The second says nothing really changed, it is just another API call, review it like any third-party dependency. The second instinct is the more expensive of the two, because it ships.

Here is the position worth holding. The discipline does not change: model the system, enumerate threats against its elements, map controls, decide what you accept. STRIDE still works, the data flow diagram is still the anchor, and a team that runs design reviews well already has most of what it needs. What changed is where the trust boundary sits. In a classic application, untrusted input crosses a boundary at the edge and becomes inert data on the other side; parameterized queries and output encoding make the separation between data and instructions structural. In an LLM application that boundary moved inside the system, and the separation stopped being structural. Your instructions and an attacker's arrive in the same channel, in the same language, and the component that reads them has no reliable way to tell which is which.

What changes when the system contains a model

Three shifts do most of the work. Everything else in AI threat modeling is downstream of them.

Natural language became an input channel that carries instructions, not just data. Prompt injection is treated as a parsing bug, and that framing produces bad controls: teams write regexes for "ignore previous instructions" and declare the problem handled. It is a confused deputy problem. The model is a deputy acting with your application's authority, and any text that reaches its context can attempt to redirect that authority: the user's message, a retrieved document, a tool's response, the contents of a PDF someone uploaded, the alt text on an image. There is no parameterized query for English. You can raise the cost of injection with instruction hierarchies, delimiters, and classifiers, but you cannot design the channel so injection is impossible, which means the control has to live somewhere other than the input.

The model and its context are assets in their own right. A classic threat model has processes, data stores, and flows. An AI system adds a set of things attackers want that were not on the old diagram: the system prompt (which encodes business logic and often the guardrails themselves), the retrieval corpus, fine-tuned weights, the embedding space, and the conversation history. Each can be extracted, and each can be poisoned. A corpus that anyone can write into is an input to every future answer, and unlike a database record, poisoned context does not look wrong when you read it.

Agency turns outputs into actions. For as long as the model only produces text a human reads, the blast radius of a bad output is a bad answer. Give the same model a tool that writes to a ticketing system, sends mail, or calls an internal API, and every output becomes a potential action taken with the application's credentials. This is the single change that most alters severity, and it is the one that arrives quietly: a feature that was reviewed as a summarizer gets a tool added in a sprint that nobody flagged for review.

Read that stack as one claim: the elements are familiar, the boundaries are not. Which is exactly why the methodology survives and the question set has to change.

STRIDE, with the AI questions added

Walk the same six categories over the same diagram. For each element, ask the classic question, then ask the version of it that only makes sense once a model is in the loop.

Nothing in that table requires a new methodology, which is the point. If you are still deciding which methodology to start from, start with STRIDE for the same reason you would on any other system: it anchors threats to diagram elements, so coverage is measurable and two engineers modeling the same feature converge. Add a privacy pass wherever personal data reaches a context window, because "the model saw it" is a disclosure question and a purpose-limitation question at the same time.

The OWASP LLM Top 10 as a design review checklist

The OWASP Top 10 for LLM Applications is the most useful public artifact in this space, and it is usually used badly: read once, quoted in a slide, never turned into a question anyone has to answer. Treat it instead as a design review checklist, where each entry is a decision the team records before the feature ships. The value is not the taxonomy. It is that ten specific decisions get made deliberately rather than by default.

One decision the list does not make for you belongs on the same page. Ask whether the prompt, the retrieved chunks, the model version, and every tool call are recorded well enough to reconstruct a decision months later. That is the repudiation row from the table above, and it is the control teams most often discover they needed after an incident rather than before one.

The official list carries the detail behind each entry and is worth reading in full before you adapt this. When you want the adversary's view rather than the defender's, MITRE ATLAS catalogs real tactics and techniques against AI systems in the ATT&CK format, which is the better source when you need to argue that a threat is realistic rather than theoretical.

A worked example: a RAG assistant

Take the system most teams are actually shipping: an internal assistant that answers questions from company documents. The data flow diagram has six elements, and every one of them is ordinary. A user, an external entity, sends a question. A chat orchestrator, a process you own, embeds the question, retrieves matching chunks, assembles a prompt, and decides what to do with the result. A model API, a process in someone else's trust zone, generates the answer. A vector store holds the chunks and their embeddings. A document ingestion pipeline, a process, reads from source repositories (the wiki, the ticketing system, the shared drive) and writes into the vector store. Those repositories are external entities, and they are the element teams forget to draw.

Walk STRIDE on three of those elements and the shape of the work becomes clear.

The ingestion pipeline: tampering, via poisoning. Anyone who can write into a source repository can write into the model's context. A customer-facing ticket queue is a public write endpoint to your assistant's knowledge, and text placed there is retrieved and treated as authoritative by construction. The control is provenance: know the source of every chunk, label it with the trust level of that source, and treat content from externally writable systems as untrusted by policy rather than by hope. Do not index what you would not repeat, and carry the source label all the way to the answer so a reviewer can see what a claim rests on.

The model API boundary: information disclosure, via context. The prompt is a data flow leaving your trust zone, and it contains whatever retrieval put in it. Two failures compound here. Retrieval typically runs with the application's permissions rather than the user's, so the corpus can answer questions the asker had no right to ask; and the assembled context may carry personal data across a boundary that your records of processing never described. The control is context minimization plus output filtering: retrieve with the caller's entitlements, not the service's; redact before the boundary rather than after; and check the response against the same sensitivity labels on the way back.

The orchestrator: elevation of privilege, via injection-driven tool calls. The orchestrator is where a generated string turns into an action, so it is where injected instructions cash out. The control is not better prompting. It is an allow-list of tools per feature, arguments validated against a schema the model does not control, and human approval on anything that writes, spends, or sends. Approval is the control that holds when the prompt-level defenses fail, and prompt-level defenses fail.

Agentic systems: model the tools, not the model

When the system becomes agentic, teams reach for a model-centric threat model: which model, how aligned, how well guardrailed. That is the wrong unit of analysis. An agent is a loop over three things, a model, a tool set, and memory, and the interesting security properties live in the last two.

The useful unit is the tool call. For each tool, ask what it can read, what it can write, what it can spend, and what happens when it is called with arguments an attacker chose. That question is answerable at design time, it does not require any claim about model behavior, and it survives a model upgrade. The corollary is uncomfortable and worth stating plainly: an agent's real privilege level is the union of its tools' permissions, evaluated as if the attacker were driving. Five tools that each look reasonable can compose into a capability nobody approved, and the classic shape is one tool that reads sensitive data and one that can reach the network.

Memory deserves its own line on the diagram. Conversation history, scratchpads, and long-term stores are data stores that the agent reads back as if it were its own reasoning, which means poisoning persists across sessions and can cross users where memory is shared. Treat memory as an untrusted input on every read, scope it per user and per tenant, and set an expiry, for the same reason you would not let a cache live forever.

Running this as a process, not a one-off

None of this is a special AI workflow. It is your security design review with a different question set attached to the same diagram, and it belongs in the same intake, the same tiering, and the same sign-off. The one thing worth adding is a trigger: re-model whenever agency expands. A new tool, a new source in the retrieval corpus, a new model with different capabilities, or a move from suggestion to autonomous action all change the trust boundaries, and a model whose boundaries have changed is not a model anymore. If scheduling a session per change sounds impossible, it is, which is the argument for running this as an async loop instead of a workshop.

This is the work our threat modeling tool is built for: threats anchored to the actual elements of a live diagram rather than listed in a document, mapped to controls from your catalog, with unmitigated counts you can see per system instead of inferring. Our own AI threat modeling studio drafts elements, candidate threats, and control mappings for a human to accept or reject batch by batch; it runs on the model you bring, every write is approval-gated, and people make the decisions that matter. We would rather be judged on that division of labor than on any claim about autonomy.

The systems are new. The reason models get skipped is not. Teams skip them when the process costs more than the shipping pressure will bear, and AI features arrive under more pressure than anything else in the backlog. Keep the question set short, anchor it to the diagram, and run it on everything.