GMD AutomationVisit Website →
← Back to blog

Manual First LLM Red Teaming for Engineers with SARIF and CVSS v4.0

September 18, 2026
Manual First LLM Red Teaming for Engineers with SARIF and CVSS v4.0

LLM red teaming is the practice of deliberately attacking a language model and its surrounding application to find failures before real attackers do. The recommended approach for most teams is manual, exploratory testing first, run by someone with a genuinely adversarial mindset, followed by automated regression testing wired into CI/CD. Map every finding to the OWASP LLM Top 10, score severity with CVSS v4.0, and report results in SARIF so they slot into tools your security team already trusts.

TL;DR:

  • Manually probing LLMs first reveals creative attack paths and context-dependent failures that automated tools may miss, especially in complex, multi-turn interactions.
  • Red teaming must evaluate the entire system surface, including prompts, data sources, plugins, memory, and downstream actions, not just the model itself.
  • Regular, automated regression testing of confirmed vulnerabilities, with detailed reproducible traces and business-impact scoring, ensures timely detection of regressions and overlooked risks.
  • Scoring findings with severity and impact, and integrating them into ongoing risk management, is essential for prioritization and meeting regulatory requirements.
  • Combining manual discovery with continuous automation creates a scalable, effective approach to securing AI systems against probabilistic and multi-faceted threats.
Gmdautomation
Build More Resilient AI Systems
Gmdautomation helps UK businesses deploy scalable AI systems designed for security, compliance, performance, and ongoing optimization.
Explore AI automation

Table of Contents

What does LLM red teaming actually test?

Red teaming an LLM is not the same job as a traditional web application pentest. A pentester usually hunts for deterministic bugs: a missing auth check, an unpatched library, a SQL injection that either works or does not. LLM security testing has to deal with probabilistic behaviour. The same jailbreak prompt might succeed three times out of ten, fail on a retry, and succeed again after a model update. That non-determinism changes how you plan, sample, and report findings, a distinction worth internalising before you write a single test case, as the LLM red teaming test plan from Phixe sets out clearly.

Scope matters as much as method. Most red-team engagements fail because they only test the model in isolation and ignore everything wired around it. You need to cover the full attack surface:

  • The system prompt and instruction hierarchy, where role confusion and instruction override attacks live.
  • Retrieval-augmented generation (RAG) data sources, which attackers can poison indirectly through documents the model later ingests.
  • Tools and plugins, where a model tricked into calling a function with attacker-controlled arguments becomes a remote execution problem, not a chatbot problem.
  • Memory and session state, which can leak one user's data into another user's context if isolation is weak.
  • Downstream sinks, meaning anywhere the model's output gets executed, rendered, or acted on without a human checking it first, such as an email sender or a database write.

Scope each test by privilege level. A customer-facing chatbot with read-only access to a knowledge base carries a very different risk profile from an internal agent with write access to a CRM. Test the second far harder than the first.

Why does LLM red teaming matter for the business?

The cost of skipping this work rarely shows up as a headline breach. It shows up as leaked customer records pulled from a RAG index, a compliance incident when a model gives regulated advice it was never authorised to give, or a "denial of wallet" attack where someone loops your agent into thousands of unnecessary API calls until the bill arrives. Each of these is a business outcome, not an abstract security metric, and each one lands on someone's desk with a dollar figure attached.

Regulatory pressure is catching up fast. The NIST AI Risk Management Framework gives US and internationally-minded teams a structured way to categorise and govern AI risk, and the EU AI Act introduces binding obligations for higher-risk AI systems that will push red-team evidence from "nice to have" into "audit requirement" for many organisations. A risk-based guide such as RAND's AI Security Guide and Risk Assessment Tool is useful here: it helps teams prioritise safeguards by mapping risk to system context rather than applying the same controls everywhere regardless of stakes.

Red-team findings only earn their value once they get converted into action. That means:

  • Tagging every finding with a severity score and a business-impact statement, not just a technical description.
  • Feeding high-severity findings straight into the next sprint, not a backlog nobody revisits.
  • Turning every confirmed vulnerability into a regression test that runs automatically, so a fix that gets reverted six months later gets caught immediately.

Pro Tip: Treat every red-team finding as a data point for the risk register before you treat it as a bug ticket. A prompt injection that lets a support bot leak a phone number is a privacy incident waiting to happen, and framing it that way gets it prioritised faster than "model said something odd".

How do vulnerabilities map to the OWASP LLM Top 10?

The OWASP LLM Top 10 gives red teamers a shared vocabulary. Instead of writing "the model did something weird when we asked it about competitors," you write "LLM01: Prompt Injection, reproducible in 6 of 10 attempts, severity high." Here is how the highest-priority categories tend to show up in real testing:

  1. Prompt injection (LLM01). An attacker embeds instructions inside user input, a document, or a webpage the model reads, overriding the system prompt. Evidence to capture: the exact injected string, the model's full response, and whether any downstream tool call fired as a result.

  2. Sensitive information disclosure (LLM02). The model repeats training data, leaks another user's context, or exposes system prompt contents when asked the right way. Capture the triggering prompt, the leaked content (redacted in your report), and how many retries it took.

  3. Supply chain vulnerabilities (LLM03). A compromised fine-tuning dataset, a poisoned plugin, or an unvetted third-party model introduces backdoored behaviour. Evidence here often means auditing the provenance of every component feeding the pipeline, not a single prompt trace.

  4. Data and model poisoning (LLM04). Similar to supply chain risk but focused on training or RAG data specifically manipulated to bias outputs. Reproducing this usually means demonstrating that a planted document changes model behaviour on unrelated queries.

  5. Improper output handling (LLM05). The model generates output that gets executed or rendered unsafely, such as unescaped HTML or a shell command passed straight to a subprocess. This is where LLM vulnerability testing overlaps most with classic web security.

  6. Excessive agency (LLM06, formerly excessive agency under different numbering across OWASP revisions). A tool-using agent takes an action beyond its intended scope, such as deleting records when it was only supposed to read them.

For every one of these, the artefact that matters is a reproducible trace: the exact input, the model version, the full output, and, where relevant, the tool-call arguments and retrieval hits. Where the attack chains multiple steps, mapping it to MITRE ATLAS tactics alongside the OWASP ID gives security teams a second reference point they may already use in existing threat models, a pairing recommended in Phixe's OWASP mapping guide.

Should you red team manually, automatically, or both?

Both, run in that order. NVIDIA describes the ideal LLM red teamer as having an "alchemist mindset": someone who probes creatively, follows a hunch, and tries the weird combination of instructions nobody documented, because that is exactly how NVIDIA defines LLM red teaming in its own technical guidance. Automation cannot replicate that instinct. A scripted fuzzer will happily run ten thousand variations of a known jailbreak template, but it will not notice that switching the conversation to a fictional framing halfway through unlocks a completely different failure mode.

Manual and automated model testing paths

That is precisely why Microsoft's Foundry red-teaming guidance recommends an initial manual round before any systematic measurement. Manual exploration maps the actual risk surface of your specific application, its prompts, its tools, its data, and that map tells you what automated probes are worth writing.

Automated red teaming earns its place once you know what you are looking for:

  • It scales to thousands of prompt variations overnight, which no human team can match.
  • It catches regressions reliably, flagging the moment a fix silently breaks after a model or prompt update.
  • It struggles with multi-turn, contextual attacks that depend on conversational build-up rather than a single malicious input.
  • Self-play attacker agents can expand coverage further, but they risk overfitting to the constraints they were given, so they work best once human testers have already established realistic attacker behaviour, a caveat raised in recent research on automated red-team agents.

The hybrid pattern that works in practice: run manual sessions to discover novel failures, turn every confirmed exploit into an automated test case, then let automation monitor for regressions continuously while your human testers move on to probing the next feature release.

What is a practical red-team workflow from planning to CI/CD?

A red-team engagement without a written plan turns into an unstructured chat session that nobody can repeat. Structure it in five stages:

  1. Plan. Define scope (which components, which privilege levels), build a harm catalogue specific to your application (what would actually hurt: data leak, financial loss, reputational damage), agree ethics and rules of engagement, and set measurable acceptance criteria before anyone writes a single adversarial prompt.
  2. Run. Execute manual sessions first, then automated probes. Because failures are probabilistic, sample each attack multiple times, not once, and record the success rate across repeats rather than a binary pass or fail.
  3. Triage. Score each confirmed finding. CVSS v4.0 gives you a defensible base score; a prompt injection that leads to full data exfiltration scores very differently from one that only produces an off-brand tone of voice.
  4. Remediate. Fix the root cause where possible (better system prompt isolation, input sanitisation, tool permission scoping), then write a regression test that encodes the exploit so it never silently reappears.
  5. Regress. Run that regression test on every relevant deployment going forward, not just once after the fix ships.

Your pre-engagement checklist should cover:

  • Explicit scope boundaries (which model versions, environments, and integrations are in play).
  • A documented harm catalogue tied to your actual product, not a generic list.
  • Ethics and disclosure rules, following coordinated vulnerability disclosure practice as outlined by CISA, so findings get fixed before they get talked about publicly.
  • Defined success metrics: attack success rate, time to detection, time to remediation.

For CI/CD integration, run automated probes as a gating step on every pull request that touches the system prompt, retrieval pipeline, or tool definitions. Output results in SARIF so they appear alongside your existing static analysis findings in whatever code review tool you already use, and fail the build when a previously fixed exploit resurfaces. Track coverage over time: how many OWASP LLM Top 10 categories does your probe suite actually exercise, and which ones are you still testing manually only.

Which tools and reporting formats should you actually use?

Three named resources cover most of what a working red team needs. The OWASP GenAI Security Project publishes the LLM Top 10 taxonomy itself, the reference point for classifying almost every finding you generate. llmsectest is a pytest-native framework built specifically around that taxonomy: it runs real probes against your model, produces SARIF, HTML, and JSON reports, and attaches a CVSS v4.0 base score to each finding automatically, which removes a lot of manual scoring guesswork. NVIDIA's garak approaches the same problem from a broader scanning angle, probing for known jailbreak patterns, data leakage, and toxic output across a wide range of attack categories.

For seed material, curated corpora save you from writing every adversarial prompt from scratch. JailbreakBench and AdvBench both provide behaviour sets specifically designed to probe refusal and bypass behaviour, useful both as manual inspiration and as automated scan inputs. Run the same corpus against a model that has been deliberately hardened to refuse too aggressively, and you get a second, easily overlooked metric: the over-refusal rate, how often the model wrongly blocks a legitimate request. A security suite that only measures attack success without measuring over-refusal will push teams toward models that are technically safer and practically useless.

A usable report needs four things every time: a reproducible trace covering the exact input, model version, and any retrieval or tool-call data; a measured success rate across repeated attempts rather than a single run; a CVSS v4.0 base score or equivalent severity rating; and a regression test that can be dropped straight into an automated suite. Token-cost figures for each probe are worth tracking too, since automated scanning at scale against a paid API can get expensive fast if nobody is watching the bill.

Essential fields in an LLM red team report

How do you embed red teaming into ongoing AI risk management?

A one-off red-team exercise before launch tells you almost nothing about the system six months later, once the model has been updated, the RAG index has grown, and three new tools have been bolted on. Red teaming needs to become a recurring line item in your AI governance process, not a box ticked once at launch, which is the logic behind RAND's risk-based security guide: align controls to context across design, development, deployment, and operation, and revisit that alignment every time the context changes.

Build this into procurement and audit readiness with a short checklist:

  • Re-run the full probe suite after every model version change, not just after application changes.
  • Track coverage against the OWASP LLM Top 10 over time, and flag categories with no recent test evidence.
  • Ask any AI vendor for their own red-team evidence and reproducible traces before signing, questions worth formalising with a vendor procurement checklist.
  • Fold findings into a formal AI model risk management process that regulators can actually audit.

Some organisations run this entirely in-house. Others reach a point where the volume of AI systems in production outpaces the security team's bandwidth to test and monitor them all continuously, and that is where a managed partner becomes worth considering rather than a forced sale.

What rookie mistakes cost teams the most time?

The biggest mistake in LLM adversarial testing is treating a single corpus hit as proof of a vulnerability. If a jailbreak prompt from a public dataset works once, that tells you almost nothing about real-world exploitability until you have run it a dozen times and recorded the actual success rate. Non-determinism punishes anyone who reports a one-off result as a confirmed finding.

The second mistake is ignoring downstream sinks. Teams pour effort into jailbreaking the model itself and forget to check what happens when its output reaches a database write, an email send, or a rendered web page. That is usually where the real damage sits, not in the chat window.

The third is poor reproducibility. A finding with no recorded model version, no saved trace, and no retry count is useless to whoever has to fix it next month.

What works instead: preserve every trace exactly as it happened, translate each finding into business-impact language before it reaches a ticket queue, keep every confirmed exploit alive as a regression test, and measure over-refusal alongside attack success so a fix doesn't quietly make the product unusable. Red teaming that only counts successful attacks and never checks what got broken in the process is only telling half the story.

— Ravi

Where should you go for primary standards and tools?

For framework mapping, start with the OWASP LLM Top 10, the taxonomy most vulnerability reports should reference by ID. For implementation guidance on structuring an engagement, Microsoft's Foundry red-teaming documentation and NVIDIA's definition of LLM red teaming both cover the manual-first, automated-for-scale approach in depth. For a ready-to-run automated framework, llmsectest on PyPI produces SARIF and CVSS v4.0 output out of the box. For governance and risk prioritisation across the AI system lifecycle, RAND's AI Security Guide is the strongest starting point.

Getting from findings to fixes without the overhead

Running a rigorous red-team programme, triaging every finding, and keeping regression tests current is genuinely demanding work, and most engineering teams already have a backlog before AI security testing joins the queue. A managed service provider can build and manage AI systems under a monthly subscription that covers implementation, ongoing operation, and security maintenance, so remediation and monitoring do not become a second full-time job for your team. Services such as Your AI answers, qualifies and books, offered under a monthly subscription, and Your AI credit controller for lettings, also subscription-based, are built and maintained with UK GDPR compliance and ongoing optimisation included, with flexible scaling if your risk profile or usage changes. If your team is weighing whether to build red-teaming and remediation capacity in-house or hand the operational load to a partner, Gmdautomation is worth a conversation before you commit engineering headcount to it.

Sources

FAQ

Which AI is best for red teaming?

No single model or tool wins outright. The strongest results come from combining a human tester's creative, "alchemist mindset" probing with an automated framework like llmsectest or garak for coverage and regression testing.

How much do red teamers get paid?

Pay varies widely by seniority, region, and whether the role covers general security testing or specialised AI red teaming, and no single reliable industry-wide figure exists. Roles that combine ML expertise with adversarial security skills tend to command a premium over general application security roles.

What does "red teaming" mean?

Red teaming means simulating a real attacker to find weaknesses in a system before someone with bad intent does. For LLMs, that means testing the model, its prompts, its data sources, and every tool it can call, not just the chat interface.

Is SOC blue or red team?

A Security Operations Centre (SOC) is generally blue team work: it defends, monitors, and responds to attacks in real time. Red teaming is the opposite role, actively attacking systems to expose weaknesses the blue team then has to defend against.