Prompt-injection testing for AI apps

Create a “Prompt-injection test plan for web apps that use AI” article that engineering teams can run in 2026 by starting from the part attackers care about, control of the prompt and the outcomes it drives. Did you know that OWASP LLM01 defines prompt injection as manipulating LLMs via crafted inputs can lead to unauthorized access, data breaches, and compromised decision-making?

Key takeaways

What you build

Why it matters

Where it shows up in the plan

A threat model you can update

AI behaviour and authority can shift between deployments without visible architecture change

Update triggers, asset inventory, assumptions log

Instrumentation that produces evidence

Chat text alone is not enough to prove impact

Prompt context traces, tool calls, policy gates, output sinks

Test cases mapped to OWASP-style categories

Coverage needs to include disclosure and downstream effects

Direct and indirect prompt injection, output handling, sensitive disclosure

Exploit evidence with reproducible steps

Engineering needs a trace they can replay and fix

Seed payloads, request parameters, model settings, observed effects

Fix guidance that closes the loop

Remediation must be verified, not just proposed

Concrete code or config changes, retest criteria

Non-destructive boundaries

You need proof without harming production or data

Proof-only exploitation and controlled blast radius

For the structure of our own engagements and evidence-driven reporting, see methodology phases, recon to report to re-test.

Did You Know?

LLM01 prompt injection is defined as manipulating LLMs via crafted inputs that can lead to unauthorized access, data breaches, and compromised decision-making.

Source: OWASP Top 10 for Large Language Model Applications

Threat model first: define the assets, the attacker paths, and evidence outcomes

To create a “Prompt-injection test plan for web apps that use AI” article, you need a threat model that is explicit about the prompt surface, the authority of the model or agent, and where outputs land in your system. Prompt injection web apps usually fail because the plan treats “the LLM response” as the product, instead of treating “the downstream effect” as the risk.

We use an engineering-friendly threat model template based on OWASP guidance for AI components, including the idea that a threat model can become stale as behaviour, authority, or effects shift between deployments.

1) Assets to list in scope

• Prompt inputs: user messages, conversation history, retrieved documents, web content, templates, system and developer instructions.
• Authority: what the model can do (tool calls, API requests, database reads, workflow triggers, file access).
• Secrets: API keys, user data, internal metadata, session identifiers, tenancy identifiers, permissions claims.
• Output sinks: rendered HTML, JSON returned to the browser, actions executed server-side, database writes, email or ticket creation.

2) Trust boundaries that matter for prompt injection web apps

• User-controlled text entering the prompt template.
• Indirect content entering the prompt via retrieval (RAG) or content ingestion.
• Tool output that is fed back into the model as context (tool output can contain attacker-controlled strings).
• Authorisation context: claims included in the prompt (for example, role lists or tenant ids) must remain server-trustable.

3) Attacker goals to map to measurable outcomes

For each goal, define a passable evidence outcome. For example, do not say “stop unauthorised access”, instead say “the plan must produce a trace showing access attempt, authorisation decision, and final data returned or action executed”.

• Unauthorised access: evidence that the app returned data or performed an action outside the caller’s policy.
• Sensitive information disclosure: evidence that private data appears in the model output or the output sink.
• Downstream exploit: evidence that model output was treated as executable data in another subsystem.
• Excessive agency: evidence that the agent tried to do more than the app intended, and that enforcement blocked it.

4) Refresh points for the threat model in 2026

In 2026, model providers and agent frameworks update frequently. Your threat model must have a refresh trigger whenever you change any of the following:

• Model identity or parameters that affect behaviour (system prompt format, tool calling mode, tool set).
• Retrieval sources or preprocessing that changes what text gets inserted into prompts.
• Policy logic, authz checks, or what claims you pass into the prompt.
• Output handling, parsers, renderers, and action executors that interpret the model output.

OWASP resources for AI threat modelling are a useful reference point for keeping assumptions current: OWASP Threat Modeling Resources, AI and agentic threat modelling.

What to instrument: log the prompt path, tool path, and output sink path

To create a “Prompt-injection test plan for web apps that use AI” article that works in practice, instrumentation must capture the full causal chain. Testers need to prove where the injection entered, how it influenced the prompt, and what effect happened after output handling.

1) Instrument the prompt assembly pipeline

• Prompt context trace: store a structured representation of the final prompt sent to the model (redact secrets).
• Component provenance: label each inserted fragment (user input, system instruction, retrieved document chunk, tool result text).
• Prompt length and truncation: record which fragments were included or dropped, because truncation can remove guardrails or hide attacker instructions.

2) Instrument retrieval and ingestion for indirect prompt injections

• RAG query and retrieved chunks: store chunk ids and sources, not only the final aggregated prompt.
• Preprocessing steps: record cleaning, HTML stripping, markdown conversion, and any “prompt-like” formatting transforms.
• Allowlist or classification decisions: log when suspicious content was filtered or labelled as untrusted.

3) Instrument tool calling and agent action execution

• Tool call requests: record function name, arguments, and correlation ids.
• Authorisation checks: log policy decisions, the inputs to authorisation, and whether the check was performed server-side.
• Tool outputs: record returned values that get fed back into the model, with careful redaction.

4) Instrument output handling and sinks

• Output parser results: if you expect JSON, record parser success or failure and where validation happens.
• Renderer context: if you render model output into HTML, log escaping mode and whether sanitisation ran.
• Action execution: confirm whether model output directly triggers behaviours or only produces suggestions that require additional validation.

We also include “evidence collection” hooks in the instrumentation spec. That means capturing enough metadata to recreate an exploit attempt without using production secrets or unsafe side effects. Our methodology emphasis on prove only exploitation aligns with this approach, see recon, enumerate, prove, report, re-test.

Test cases you can run: map injection scenarios to OWASP-style categories

To create a “Prompt-injection test plan for web apps that use AI” article, you must list test cases with a clear mapping to OWASP-style categories, while keeping them executable by engineering teams. In 2026, you should expect both direct prompt injection (user message tries to override instructions) and remote or indirect prompt injection (attacker-controlled content enters via retrieval, embeds, or documents).

Use OWASP’s LLM prompt injection prevention guidance as a starting taxonomy. Reference: OWASP LLM Prompt Injection Prevention Cheat Sheet.

1) Category mapping and concrete test case patterns

Direct prompt injection (user-controlled)

• Instruction override: payload tries to replace system instructions with attacker instructions, for example “ignore the previous instructions” patterns.
• Role and tenant escalation: payload asks the model to output data for another user or different tenancy.
• Tool misuse request: payload asks for tool calls that would be blocked under your normal policy.

Remote or indirect prompt injection (retrieval and content)

• Injected document chunks: a retrieved chunk contains prompt-like directives to change behaviour.
• Indirect embedding: HTML or markdown content fetched from a URL is inserted into the prompt and contains attacker directives.
• Multi-hop poisoning: tool output gets stored and later retrieved, then injected on subsequent requests.

Improper output handling

• Schema confusion: prompt asks for JSON but uses fields that trigger unsafe downstream parsing.
• Confused deputy: output is treated as an instruction by another subsystem (for example, a workflow engine consumes it directly).
• Type mismatch: output returns strings where you expect structured data, and validation is missing.

Sensitive information disclosure

• Secrets in context: attempt to force the model to quote hidden system prompt text or internal instructions if those are mistakenly included in prompt context.
• Private user data: request that the model regurgitates other users’ records retrieved under incorrect authz.
• Membership and entitlements leakage: force output to include role lists or policy decisions that should not be revealed.

Excessive agency

• Overbroad tool calls: ask the model to call tools with arguments outside the user’s scope.
• Repeated attempts: payload asks for repeated action retries that could spam systems or bypass rate limits.
• Bypass via paraphrase: payload rephrases a disallowed action and relies on the model to convert it into a tool call.

Insecure plugin design style issues (if you support plugins or tool runners)

• Untrusted input reaches tool runtime: plugin code handles untrusted content without strict access control.
• Argument injection: attacker influences tool arguments so the tool performs privileged work.

2) A practical test case template engineering can execute

To create a “Prompt-injection test plan for web apps that use AI” article, include a repeatable template in your plan so teams can build suites quickly.

1. Name: short label that includes injection type (direct, indirect, tool-fed).
2. Threat category: OWASP-style category label.
3. Preconditions: auth state, tenant, enabled tools, retrieval settings.
4. Seed payload: the crafted prompt or injected content.
5. Expected safe behaviour: no disclosure, tool calls denied, outputs validated.
6. Evidence to capture: prompt trace, tool call log, output sink output, authorisation decision trace.
7. Fix guidance area: where engineering should look (prompt assembly, policy gate, output parser, renderer, tool runner).

3) Reducing false positives in 2026

In 2026, teams get noise because many prompt injection patterns appear in benign educational text. Some guidance from Google’s prompt injection research notes that false positives are common where suspicious phrasing appears in education material. That means your plan should include triage logic that distinguishes “instruction-like strings” from “malicious intent that changes authorisation or effects”.

• Record whether the injection changed the model’s action or output sink.
• Classify findings based on evidence of impact, not just similarity to a signature phrase.
• Use an intent classifier approach if you have it operationally, as suggested by the research describing LLM-based classification for suspicious text.

Indirect prompt injections hide inside large datasets fed directly to models.

How to produce exploit evidence that engineers can act on

To create a “Prompt-injection test plan for web apps that use AI” article, the evidence section must be built for replay. Engineering teams need the minimum set of artefacts that recreate the exploit attempt and show which enforcement gate failed.

1) Evidence artefacts to attach to each finding

• Request metadata: endpoint, method, authenticated user identity context (redacted), tenant or workspace id.
• Seed payload and content injection: the exact user prompt or injected document text.
• Model configuration: model id, temperature or other relevant parameters (redact if needed).
• Prompt trace: structured final prompt with provenance tags for each segment.
• Tool call log: tool name, arguments, correlation id, and whether authorisation was enforced.
• Output sink output: the final response body shown to user or stored internally, with redactions.

2) Evidence structure: from impact statement to proof

We use a reporting pattern like recon to enumerate to prove to report to remediate and re-test. The point is to make each finding traceable to a closure path. A sample report format can be seen at report sample structure, impact, evidence, and the fix.

Impact describes the risk in operational terms, not just “the model said something wrong”.

Evidence shows the prompt path, tool path, and output sink path that caused the effect.

The fix lists concrete changes, plus a retest criteria.

3) Safe proof guidance for prompt injection test plans

• Prefer proof-only exploitation where possible, do not dump sensitive datasets.
• Use synthetic or test accounts and seeded documents.
• Throttle test activity to avoid inducing outages or triggering safety systems.
• Redact secrets in stored prompt traces and output logs.

4) Instrumented replay: how to make the finding reproducible

Engineering teams need a replay script or a deterministic test harness. If the model is nondeterministic, record the prompt trace and model settings, then run multiple trials and store the distribution of outcomes. The evidence should still link to the enforcement points that were supposed to stop the action.

Did You Know?

LLM05 improper output handling can enable downstream security exploits, including code execution that compromises systems and exposes data.

Source: OWASP Top 10 for Large Language Model Applications

Fix guidance: how to close the gaps for prompt injection web apps

To create a “Prompt-injection test plan for web apps that use AI” article, fix guidance must specify which layer failed and what code or configuration change to apply. Prompt injection mitigations are not only about filtering phrases. They are about enforcing trust boundaries, validating outputs, and applying authorisation at the time of effect.

1) Fixes for direct prompt injection

• Separate instructions from data: treat user input as untrusted content in the prompt assembly layer.
• Constrain tool usage: build allowlists for tool names and argument shapes.
• Harden policy gates: authorisation decisions must be server-side and independent of model output.

2) Fixes for indirect prompt injection

• Untrusted retrieval handling: mark retrieved content as untrusted and do not allow it to override system-level goals.
• Structure retrieval output: if you feed documents into the model, wrap them in a clear delimitation format and remove instruction-like formatting.
• Classification and filtering: triage retrieved chunks, especially those that resemble instruction-like patterns.

3) Fixes for improper output handling

• Strict output schemas: validate model output against a schema before any downstream use.
• Defensive parsing: treat parsing failures as safe defaults, do not “best effort” execute.
• Escape and sanitise: for HTML or markdown outputs, ensure sanitisation occurs before rendering.

4) Fixes for sensitive information disclosure

• Minimise what enters the prompt: do not include secrets or internal policy text in prompt context.
• Access control on retrieval: ensure RAG queries are scoped by authz before fetching documents.
• Output redaction: apply redaction rules on the output sink, not only on the model response.

5) Fixes for excessive agency

• Capability-based controls: tools should require explicit server-side permission checks.
• Rate limiting and quotas: enforce action limits independent of model behaviour.
• Action confirmation flows: where actions are high impact, require an additional safe step not controlled by the model.

6) How to write the fix section so teams can close it

Engineering teams close issues when fixes are specific. Each fix item should include a measurable retest expectation.

• Change: what to modify (prompt assembly, policy gate, output parser, tool runner).
• Evidence linkage: reference the evidence artefacts that should change after the fix.
• Re-test criteria: list which test case(s) must no longer show impact, and what “no impact” means in logs.

When we structure report closure, we focus on the same loop you want in your prompt injection test plan: remediations followed by retest. That is how our phases are laid out, see Remediation and re-test.

Build the full suite: from recon to retest to regression

To create a “Prompt-injection test plan for web apps that use AI” article, you need a full lifecycle. Teams should start with application mapping, then run prompt injection scenarios, then verify fixes with evidence, then add regression tests.

1) Suite phases you can implement

• Recon: map endpoints, auth states, tool availability, and retrieval sources.
• Enumeration: identify prompt inputs and output sinks, identify indirect content flows.
• Exploitation (prove only): run injection test cases that attempt to change outcomes.
• Report: each finding includes evidence and concrete fixes.
• Remediate and re-test: re-run the same suite and confirm evidence changed.

2) Regression tests that keep working in 2026

Prompt injection web apps evolve. Your regression suite should include at least:

• Direct injection payloads for each user input pathway.
• Indirect injection payloads for each retrieval or content ingestion pathway.
• Output handling cases for each output parser or renderer.
• Tool calling and action execution cases for each tool.

When you add new tools or change retrieval sources, add new test cases. This avoids stale coverage, which is the same risk noted in OWASP AI and agentic threat modelling resources.

3) Where Earthshaker Security style reporting aligns with this plan

Our reporting format emphasises evidence and prescribed fixes, and our approach is non-destructive proof with a closure loop. If you want to align your internal deliverable format with this style, review our sample at report sample, evidence and the fix. This helps engineers map prompt injection test plan findings into development work items.

Frequently asked questions

What is a prompt-injection test plan for web apps that use AI, and what does it include?

A prompt-injection test plan for web apps that use AI includes a threat model, instrumentation requirements, a suite of direct and indirect injection test cases, and evidence capture rules. It also includes how you produce exploit evidence plus fix guidance that engineering teams can act on, so outcomes are verifiable in logs and output sinks.

How do I map prompt injection test cases to OWASP-style categories in 2026?

Map each test case to a category such as prompt injection, improper output handling, sensitive information disclosure, and excessive agency. Then attach evidence that shows the prompt path, tool path, and output sink path that led to the effect, not only the model response text, as described in the OWASP Top 10 for Large Language Model Applications.

What should we instrument to prove a prompt injection finding is real?

You should instrument prompt assembly with provenance, retrieval and ingestion steps for indirect content, tool calls and server-side authorisation decisions, and the output sink that receives the model output. For a prompt injection web app, this instrumentation lets you produce exploit evidence that engineering teams can replay and fix.

How do we reduce false positives when testing prompt injection web apps?

Do not treat signature-like phrasing as a vulnerability by itself. Instead, triage based on whether the injection actually changed outcomes, such as an authorisation decision, a data disclosure event, or a downstream action or parser behaviour.

Should we test only direct prompt injection, or also remote and indirect injection?

You should test direct and indirect injection. In real prompt injection web apps, attacker-controlled text often enters through retrieval, documents, or tool outputs, so indirect prompt injection test cases are essential for coverage in 2026.

What fix guidance is most actionable for engineering teams after prompt injection testing?

Actionable fix guidance specifies the layer to change and what evidence should change after remediation, including prompt assembly trust boundaries, output schema validation, renderer sanitisation, and server-side authorisation for tool actions. This is how you create a prompt-injection test plan for web apps that use AI that closes the loop with retestable criteria.

Conclusion

Create a “Prompt-injection test plan for web apps that use AI” article by starting with a threat model that defines measurable outcomes, then instrument the prompt path, tool path, and output sink path so you can produce replayable exploit evidence. Map test cases to OWASP-style categories such as LLM01 prompt injection, LLM05 improper output handling, and LLM06 sensitive information disclosure, then write fix guidance that engineering teams can implement and verify with a retest loop.

Contact us