Which AI implementations fall within the scope of NCSA AI Security Guidelines?

The guidelines encompass traditional Machine Learning models as well as Generative AI and LLM systems deployed within enterprise environments or public services.

How does security testing for LLM systems differ from standard web or API testing?

In addition to web and API vulnerabilities, testing must evaluate AI-specific threat vectors including Direct and Indirect Prompt Injection, vector database authorization bypass, and malicious agent execution.

NCSA AI Security Guidelines: lifecycle and governance

A summary of NCSA's AI Security Guidelines, covering the 7-phase secure AI lifecycle, governance, risk management, and practical testing focus areas.

Executive Summary and Mandate Overview

Target Audience AI Developers, Enterprise AI Integrators, Generative AI and LLM Deployers, and Public Sector

Mandatory Frequency Lifecycle-aligned (Phases 0 to 6. Concept to Model Decommissioning)

Required Scope LLM Prompts, Model Pipelines, Vector DBs, Training Sets, Supply-chain Dependencies

Non-Compliance Risk Exposure to Prompt Injection, Data Poisoning, Privacy Leakage and AI Governance Failures

Purpose and intended audiences

NCSA presents the document as guidance for developing, managing, and using AI systems securely and trustworthily.

It spans policy-level decision making for executives and technical practice for AI development and operations teams.
Its intended stakeholders include legal and data-protection functions, cybersecurity teams, employees, customers, data subjects, and supply-chain vendors.
The guidance draws on international standards and technical material including ISO/IEC, ENISA, and OWASP alongside Thai laws and practices.

The secure AI lifecycle framework

The guidelines structure security controls across the full lifecycle, from pre-investment planning to decommissioning.

Phase 0: Concept - Identify business context, legal requirements, assets, threats, vulnerabilities, and risks before development.
Phases 1-3: Secure Design, Secure Development, and Secure Verification.
Phases 4-6: Secure Deployment, Secure Operations and Maintenance, and Proper Disposal.

Governance, risk, and accountability

Governance uses the GRC framework and stresses that risk must be evaluated in each organization's specific context.

Document security-related roles, authority, and accountability for the AI system.
Integrate AI risk into enterprise risk management and track relevant Thai laws and regulations.
Establish auditing, certification, communication, and training for controllable, auditable operations.

Practical implementation focus

The document recommends key measures to support auditing and retrospective review.

Establish Data Provenance and vulnerability scanning for processing software before launch.
Test both Direct and Indirect Prompt Injection; evaluate RAG Vector DB access controls to prevent role-based leakage.
Conduct AI Red Teaming and penetration testing across models, APIs, and surrounding software.

Requirements and Testing Scope Matrix

Summary of the referenced clauses, the testing scope they cover, and the expected evaluation cycle.

Reference	Mandate Title	Scope Required	Testing Cycle
Phases 0 to 3 (NCSA AI)	Concept, Design, Data Preparation, and Model Training Risk Governance	Training Datasets, Data Poisoning Defense, Model Supply Chain	Pre-development lifecycle stage
Phases 4 to 6 (NCSA AI)	Pre-deployment Verification, LLM Threat Monitoring, and Model Decommissioning	Prompt Injection, RAG Vector DBs, LLM APIs, System Integration	Pre-launch and continuous lifecycle

Compliance Readiness Self-Assessment

Select items your organization has completed to evaluate your readiness score.

Direct and Indirect Prompt Injection testing conducted on LLM-integrated applications RAG Vector Databases configured with role-based access controls to prevent cross-tenant data leakage Training datasets audited for provenance and integrity to prevent Data Poisoning AI Red Teaming and Penetration Testing executed across models, APIs, and surrounding infrastructure prior to launch

STH AI SECURITY TESTING

LLM-integrated App Pentest and AI Red Teaming

An LLM-integrated application has a wider attack surface than the model alone. The application, input paths, RAG or vector store, agent tools, identity, data, and event logging must be assessed together.

LLM-integrated application pentest

Validate controls at the system integration points, rather than limiting the assessment to model behaviour.

User authorization, tenant and data separation, and data exposure through responses or retrieved documents.
RAG, vector store, model-provider, API, and third-party tool integrations, including token and secret boundaries.
Approval and authorization enforcement before an agent invokes a tool or performs an impactful action.

AI Red Teaming

Simulate adversarial behaviour within an authorized scope to demonstrate whether guardrails and protective workflows work in practice.

Prompt injection and indirect prompt injection through documents or data used as context.
Jailbreaking or attempts to move the system outside its intended purpose, with validation of unsafe-output handling.
Agentic-AI risks where a prompt or tool invocation could affect data, accounts, or downstream systems.

Expected testing evidence

An authorized scope, test environment and data, stop criteria, and emergency contacts.
Evidence, severity, impact conditions, and remediation guidance that can be independently retested.
A retest after remediation and a record of controls or risks that need production monitoring.

NCSA's Secure Verification phase gives active testing, LLM penetration testing for prompt injection and jailbreaking, data-leakage risks, and dangerous agent actions as examples of evidence to verify before release.

Talk to STH about LLM-integrated App Pentest and AI Red Teaming