Siam Thanat Hack Co., Ltd.

AI Red Teaming: Adversarial Testing Guide for AI Models, RAG Systems, and Autonomous Agents

A technical security guide grounded in the OWASP GenAI Red Teaming Manual v1.0 (October 2026), mapping 6 GenAI lifecycle stages, 3 access postures, an 8-phase testing methodology, 13 Vendor Evaluation Criteria (v1.0), 7-layer MAESTRO threat modeling, and MCP hardening, paired with practical alignment to NIST AI 100-2e2025 (March 2025 Final), NIST AI 600-1 (MS-2.7-007), Google SAIF 2.0 (Assurance: Red Teaming and SAIF Focus on Agents), and Thai regulatory guidance.

Overview
Target Audience
Security engineering teams, AI developers, and enterprises deploying RAG, LLM applications, or autonomous agents
Testing and Review Frequency
Prior to production deployment and periodically upon major model, RAG data, or agent permission updates, per STH recommendations
Assessment Scope
Model evaluation, guardrails verification, RAG and vector databases, autonomous agent workflows, and MCP server integrations under OWASP Manual v1.0, NIST AI 100-2e2025, and Google SAIF 2.0
Relevant Risks
Prompt injection, training and RAG data poisoning, sensitive model data disclosure, agent rogue actions, and tool hijacking

Four Architectural Testing Layers Derived from the OWASP GenAI Red Teaming Manual

The OWASP GenAI Red Teaming Manual v1.0 (October 2026) organizes testing across 6 lifecycle stages, 3 access postures (black-box, gray-box, and white-box), and technical attack sections 4.2 to 4.7, which can be scoped operationally across four architectural layers alongside NIST and Google SAIF.

Four architectural layers for AI red teaming: Base Model Layer (4.2), Guardrails and Classifiers (4.7), RAG and Infrastructure (4.3, 4.4, 4.6), Runtime Agentic and MCP (4.5)
Four architectural layers for AI red teaming

AI red teaming under the OWASP GenAI Red Teaming Manual v1.0 spans the base model, guardrails, RAG pipelines, and runtime agentic or MCP workflows.

  • Base Model Layer (4.2)
    Assess training-data leakage and alignment robustness
  • Guardrails and Classifiers (4.7)
    Test input filters, output filters, and safety classifier bypasses
  • RAG and Infrastructure (4.3, 4.4, 4.6)
    Evaluate vector databases, multimodal inputs, and resource exhaustion
  • Runtime Agentic and MCP (4.5)
    Test tool permissions, MCP servers, and multi-agent execution flows
  • Base Model Layer (Direct AI Vulnerabilities Section 4.2 and Model Selection stage) probes foundation model behavior, training-data leakage risks, alignment robustness, and model-recovery exposures.
  • Guardrails and Classifiers Layer (Guardrail & Safety-Classifier Evasion Section 4.7) evaluates safety classifiers, system prompt filters, and token-level input and output defenses against evasion.
  • RAG and Infrastructure Layer (AI Pathway Vulnerabilities 4.3, Multimodal 4.4, and Resource-Exhaustion 4.6) examines retrieval-augmented generation pipelines, vector databases, orchestration middleware, and non-text inputs.
  • Runtime Agentic and Ecosystem Layer (Agentic System Attack Vectors 4.5 and Deployment or Continuous Improvement stages) assesses autonomous agent decision-making, tool permissions, Model Context Protocol (MCP) servers, and multi-agent coordination.
  • Practical Alignment with NIST Adversarial ML Taxonomy and Google SAIF connects offensive test scopes to NIST AI 100-2e2025 (March 2025 Final), covering both Predictive AI attacks (evasion, poisoning, and privacy) and Generative AI attacks (supply-chain compromise, direct and indirect prompt injection, and agent security), alongside NIST AI 600-1 action MS-2.7-007 and Google SAIF 2.0 Red Teaming and SAIF Focus on Agents controls targeting Sensitive Data Disclosure and Rogue Action (for technical scoping rather than an official joint mapping).

Eight-Phase Offensive AI Red Teaming Methodology

Offensive AI assessments follow an 8-phase lifecycle from governance scoping to automated regression testing, ensuring repeatable and actionable findings.

Eight-phase offensive AI red teaming lifecycle: Phases 1 to 2 Plan and Recon → Phases 3 to 4 Surface Map and Exploit → Phases 5 to 6 Persist and Post-Exploit → Phases 7 to 8 Report and Remediate
Eight-phase offensive AI red teaming lifecycle

Repeatable AI red teaming links rules of engagement to measurable test oracles, blue-team detections, and automated regression suites.

  1. Phases 1 to 2 Plan and Recon
    Set rules of engagement, access posture, and model supply-chain mapping
  2. Phases 3 to 4 Surface Map and Exploit
    Map trust boundaries and test prompt injection and multimodal inputs
  3. Phases 5 to 6 Persist and Post-Exploit
    Test memory poisoning, tool privilege escalation, and blast radius
  4. Phases 7 to 8 Report and Remediate
    Document reproducible test oracles, detections, and regression suites
  • Phase 1 Planning and Scoping aligns stakeholders, establishes access models across black-box, gray-box, and white-box postures, and defines rules of engagement with AI governance teams.
  • Phase 2 Reconnaissance and Fingerprinting identifies public model artifacts, supply-chain provenance, model architectures, and deployed libraries.
  • Phase 3 Surface Mapping establishes trust boundaries, maps data flows across retrieval pipelines, and overlays architectural threat models.
  • Phase 4 Exploitation and Impact tests direct and indirect prompt injection, multimodal input surfaces, guardrail evasions, and economic denial-of-sustainability scenarios.
  • Phase 5 Persistence and Escalation investigates vector database poisoning, agent memory persistence, and unauthorized tool privilege escalation.
  • Phase 6 Post-Exploitation measures blast radius, system stability degradation, and real-world business impacts from compromised autonomous agents.
  • Phase 7 Evaluation and Reporting documents findings using reproducible test oracles, structured impact scoring, and remediation recommendations.
  • Phase 8 Post-Engagement and Remediation integrates findings with defensive teams, builds detection signatures, and establishes automated regression test suites.

Enterprise Vendor Evaluation Criteria for AI Red Teaming

The OWASP Vendor Evaluation Criteria for AI Red Teaming v1.0 (January 2026) provides an enterprise procurement rubric spanning 13 criteria (Sections 3.1 to 3.13) plus Green Flag and Red Flag indicators across system tiers.

Enterprise vendor evaluation criteria for AI red teaming: Classify System Complexity, Technical and Threat Realism (3.1 to 3.4), Evaluation Rigor and Oracles (3.5, 3.8), Data Governance and Ethics (3.7, 3.13)
Enterprise vendor evaluation criteria for AI red teaming

Assess green and red flags across all 13 OWASP criteria (3.1 to 3.13) according to whether the system is simple GenAI or advanced agentic AI.

  • Classify System Complexity
    Separate simple prompt-response systems from advanced agentic architectures
  • Technical and Threat Realism (3.1 to 3.4)
    Assess adversarial ML depth and domain-specific threat modeling
  • Evaluation Rigor and Oracles (3.5, 3.8)
    Require reproducible parameters, test oracles, and clear retesting
  • Data Governance and Ethics (3.7, 3.13)
    Protect confidential prompts and block unauthorized external API egress
  • Distinguishes Simple GenAI Systems with single prompt-response interactions from Advanced GenAI Systems featuring agent workflows, tool execution, memory, and external APIs.
  • Technical Competence (3.1) and Methodology & Coverage (3.2) evaluate assessor proficiency in adversarial machine learning, LLM and agent internals, and structured attack coverage.
  • Adversarial Creativity & Domain Expertise (3.3) and Realism of Threat Modeling (3.4) assess the ability to synthesize novel domain-specific attack paths beyond static prompt lists.
  • Evaluation Rigor & Metrics (3.5) and Transparency & Explainability (3.8) require reproducible test oracles, clear parameter documentation, and verifiable remediation retesting.
  • Data Governance & Security (3.7) and Legal, Ethical, and Compliance Posture (3.13) protect confidential prompts and client intellectual property against unauthorized third-party API transmission.

Multi-Agent System Threat Modeling with MAESTRO and MCP Security

Securing multi-agent architectures combines the 7-layer MAESTRO framework in OWASP's Multi-Agentic System Threat Modelling Guide v1.0 with OWASP MCP development and third-party server controls.

Seven-layer MAESTRO threat model and MCP controls: 7-Layer MAESTRO Architecture, Stop Tool Poisoning and Rug-Pulls, Least Privilege and HITL Gates, Transport Sandbox and Schemas
Seven-layer MAESTRO threat model and MCP controls

Securing multi-agent pipelines pairs the 7-layer MAESTRO model with signed MCP manifests, OAuth 2.1 scopes, version pinning, and transport sandboxing.

  • 7-Layer MAESTRO Architecture
    Map threats from foundation models up to the agent ecosystem alongside vertical security
  • Stop Tool Poisoning and Rug-Pulls
    Inspect tool metadata, sign manifests, and pin approved server versions
  • Least Privilege and HITL Gates
    Enforce short-lived OAuth 2.1 scoped tokens and human approval for risky calls
  • Transport Sandbox and Schemas
    Isolate STDIO and Streamable HTTP transports and validate input-output schemas
  • Maps threats across all seven MAESTRO layers: Foundation Models (Layer 1), Data Operations (Layer 2), Agent Frameworks (Layer 3), Deployment Infrastructure (Layer 4), Evaluation & Observability (Layer 5), Security and Compliance (vertical Layer 6), and Agent Ecosystem (Layer 7).
  • Mitigates tool poisoning by evaluating hidden prompt instructions inside tool descriptions and verifying cryptographic tool manifests against runtime behavior.
  • Enforces least-privilege agent execution through short-lived scoped tokens (OAuth 2.1 or OIDC), centralized policy enforcement such as policy-as-code, and Human-in-the-Loop (HITL) approvals for high-risk calls.
  • Defends against post-approval tool rug-pulls using version pinning and manifest checksum verification alongside vetting of third-party MCP tool servers.
  • Hardens transport boundaries across STDIO or Unix sockets for local servers and Streamable HTTP for remote servers, validating schemas on every tool input and output.

Applying AI Red Teaming to Thai Regulatory and Governance Guidance

Thai sector regulators publish policy guidelines, principle-based frameworks, and Soft Regulation guidance whose technical assurance goals can be supported with structured AI red teaming evidence.

Mapping AI red teaming evidence to Thai governance guidance: NCSA AI Guidelines B.E. 2568, BOT Circular Wor 5994/2568, SEC AI Framework (Aug 2569), OIC AI Guidelines (Nov 2568)
Mapping AI red teaming evidence to Thai governance guidance

AI Red Teaming supplies technical evidence for risk assessments under Thailand's voluntary and policy-based AI frameworks while keeping voluntary guidance distinct from statutory mandates.

  • NCSA AI Guidelines B.E. 2568
    Voluntary adversarial ML and non-AML testing across all seven lifecycle stages
  • BOT Circular Wor 5994/2568
    Policy guideline for continuous use-case-proportionate attack simulation
  • SEC AI Framework (Aug 2569)
    Clause 2.4.7(3) recommends pre-deployment and post-deployment red team testing
  • OIC AI Guidelines (Nov 2568)
    Soft Regulation Clause 2.2.2 supports risk-proportionate AI vulnerability testing
  • NCSA AI Security Guidelines B.E. 2568 (voluntary guidance): Evaluates adversarial machine learning and non-AML vulnerabilities across all seven lifecycle stages.
  • Bank of Thailand Wor 5994/2568 (policy guideline): Financial institutions and payment providers can apply AI red teaming to support risk monitoring and attack-pattern testing commensurate with high-risk AI use cases.
  • SEC AI Governance Framework (August 2569, hybrid principle-based framework): Directly references Red Team and Adversarial Testing before and after deployment in Clause 2.4.7(3), alongside Annex 2 impact assessment and Annex 5 example AIMS controls.
  • OIC Insurance AI Governance Guidelines (November 2568, Soft Regulation): Supports Clause 2.2.2 risk-proportionate AI security and vulnerability testing recommendations alongside Clauses 2.2.1 and 2.2.7.

References and Assessment Scope

Summary of source references, assessment scope, and review timing in the context of each item.

Reference Topic Assessment Scope Testing Cycle
OWASP GenAI Red Teaming Manual v1.0 (October 2026) Four Architectural Testing Layers (Sections 4.2–4.7 and 6 Lifecycle Stages) Assesses Base Model (Direct AI Vulnerabilities 4.2), Guardrails (4.7), RAG and Infrastructure (4.3, 4.4, 4.6), and Runtime Agentic Behavior (4.5) Prior to production deployment and upon material model, RAG, or architectural changes
OWASP GenAI Red Teaming 8-Phase Methodology Eight-Phase Offensive Testing Methodology Structured execution across Planning and Scoping (1.6 Black/Gray/White-box), Reconnaissance, Surface Mapping, Exploitation, Persistence, Post-Exploitation, Reporting, and Remediation Aligned with organizational risk assessments and major release cycles
OWASP Vendor Evaluation Criteria v1.0 (January 2026) Vendor Evaluation Criteria for AI Red Teaming Differentiates Simple vs Advanced GenAI systems and evaluates providers across 13 criteria (Sections 3.1 to 3.13) with Green Flags and Red Flags During procurement and independent assessor selection
OWASP Multi-Agentic System Threat Modeling (MAESTRO) and MCP Security Multi-Agent System Threat Modeling and MCP Security Addresses 7 MAESTRO layers, mitigates tool poisoning and rug-pulls via version pinning/checksums, scoped OAuth 2.1, HITL, and STDIO/Streamable HTTP hardening During architecture design and prior to integrating external agent tools
NIST AI 100-2e2025, AI 600-1 MS-2.7-007, and Google SAIF 2.0 Adversarial Attack Taxonomy and SAIF Focus on Agents Alignment Classify evasion, poisoning, privacy, and abuse attacks under NIST AI 100-2e2025 alongside MS-2.7-007 red-teaming evidence and SAIF Focus on Agents boundary verification Prior to release and upon material agent architecture or permission changes
Thai Governance Guidance (NCSA, BOT, SEC, OIC) Technical Assurance Evidence Supporting Thai AI Guidance Supports NCSA voluntary lifecycle guidance, BOT Wor 5994/2568 policy guidance, SEC 2569 Clause 2.4.7(3) principle-based Red Team guidance, and OIC 2568 Clause 2.2.2 Soft Regulation Scaled by use-case risk and organizational governance cycles rather than a uniform annual mandate

Implementation Checklist

Select items your organization has completed. The percentage shows selected items only and does not assess security or compliance.

0%

Frequently Asked Questions (FAQ)

Key answers and practical guidance addressing common compliance questions.

How does AI Red Teaming differ from traditional penetration testing or model evaluation?

Standard model evaluation focuses on accuracy or single-prompt toxicity, whereas AI Red Teaming emulates realistic adversaries across models, guardrail filters, RAG architectures, and autonomous agents to uncover boundary breakouts, unauthorized tool calls, and data exfiltration.

How does the OWASP GenAI Red Teaming Manual v1.0 structure testing?

Version 1.0 (October 2026) combines 3 access postures (black-box, gray-box, and white-box), 6 GenAI lifecycle stages, an 8-phase testing methodology (Phases 1 to 8), and technical attack domains in Sections 4.2 to 4.7 spanning direct AI vulnerabilities, AI/RAG pathways, multimodal surfaces, agentic vectors, resource exhaustion, and guardrail evasion.

What key criteria should enterprises look for when procuring AI Red Teaming services?

Under the OWASP Vendor Evaluation Criteria v1.0 (January 2026), enterprises distinguish Simple vs. Advanced GenAI systems and evaluate providers against 13 criteria (Sections 3.1 to 3.13) and Green/Red Flags spanning adversarial ML expertise, creativity, reproducibility, and prompt data governance.

Which Thai AI governance frameworks align with AI Red Teaming evidence?

AI Red Teaming supplies technical evidence supporting the voluntary NCSA AI Security Guidelines, BOT policy guideline Wor 5994/2568, SEC principle-based Clause 2.4.7(3) on Red Team and Adversarial Testing (August 2569), and OIC Soft Regulation Clause 2.2.2 on risk-proportionate AI security and vulnerability testing (November 2568).

How does OWASP GenAI Red Teaming relate to NIST AI RMF, NIST AI 100-2e2025, and Google SAIF 2.0?

OWASP GenAI Red Teaming Manual v1.0 supplies the operational 8-phase methodology and four-layer test domains, which organizations map to NIST AI 100-2e2025 attack and mitigation taxonomy (evasion, poisoning, privacy, and abuse), NIST AI 600-1 MS-2.7-007 red-teaming evidence, and Google SAIF 2.0 assurance and Focus on Agents boundary verification.

Official source documents

OWASP GenAI Red Teaming Manual v1.0 and Adversarial AI Testing Suite

Go to the official source

Request a quote