AI Red Team
Establish an AI red team responsible for conducting recurring, authorized, and threat-informed red-teaming exercises to identify and remediate vulnerabilities in AI-enabled systems before deployment and throughout operation. AI red-teaming simulates realistic adversary behavior to evaluate how attacks could affect the confidentiality, integrity, availability, safety, privacy, and mission performance of an AI-enabled system. Red-teaming exercises should consider the complete AI-enabled system, including models and data, agents (including memory and tools), data flows, decision processes, application logic, retrieval systems, identities and permissions, software dependencies, non-AI system components, infrastructure, user interfaces, and human workflows. An AI red team exercise can be organized into three phases: planning and scoping the exercise, executing the selected exercises, and assessing the results to guide reporting and remediation. 1. **Plan and Scope** - Document the system's intended use, deployment environment, users, sensitive data, connected resources, and potential consequences of failure or misuse. Diagram the system's components, trust boundaries, data flows, external services, human decision points, and training- and inference-time access points. - Establish rules of engagement covering authorized systems, accounts, data, techniques, test windows, resource limits, escalation procedures, evidence handling, and stop conditions. Plan destructive, privacy-invasive, or high-cost tests for isolated environments with appropriate safeguards. - Develop a threat model based on the system's operating environment and relevant adversary behavior. Define the adversary's objectives, access, knowledge, capabilities, resources, and constraints, and account for digital and physical attack paths and the role of human oversight. - Use ATLAS tactics, techniques, and procedures to identify relevant adversary behaviors and construct threat vectors. Prioritize them according to likelihood and severity of impact to the system. - Define success criteria and stopping conditions. Identify task-level metrics for effects on the AI capability and operational metrics for measuring impact to the overall system. 2. **Execute** - Conduct the selected exercises using manual and automated methods as appropriate. Automation can generate input variations, replay attack sequences, and evaluate responses at scale. Human testers can develop system-specific attacks, adapt to observed defenses, and investigate unexpected behavior. - Follow the rules of engagement and record attack activity, system responses, control behavior, deviations from the test plan, and evidence needed to evaluate the results. - Stop or escalate testing when predefined conditions are reached. After testing, remove test accounts, modified data, installed software, persistent instructions, and other exercise artifacts. 3. **Assess, Report, and Improve** - Evaluate the results against the defined task and operational metrics. Document successful and unsuccessful attacks, their consequences, observed control behavior, deviations from the test plan, and gaps in the threat model. - Report findings to the appropriate developers, defenders, operational teams, risk owners, and other stakeholders. - Assign findings to responsible owners, track remediation, and retest corrected systems. - Use demonstrated attacks to improve preventive controls, detection, incident response, and recovery. Where appropriate, convert confirmed failures into regression tests, evaluation datasets, detection logic, monitoring requirements, or deployment criteria. Red-teaming is a continuous process and should be repeated as the threat landscape evolves and when changes are made to the system, its components, intended use, or deployment environment. New threat intelligence, vulnerabilities, and test results should inform the scope and priorities of future exercises.
Confidence
Establish an AI red team responsible for conducting recurring, authorized, and threat-informed red-teaming exercises to identify and remediate vulnerabilities in AI-enabled systems before deployment and throughout operation. AI red-teaming simulates realistic adversary behavior to evaluate how attacks could affect the confidentiality, integrity, availability, safety, privacy, and mission performance of an AI-enabled system. Red-teaming exercises should consider the complete AI-enabled system, including models and data, agents (including memory and tools), data flows, decision processes, application logic, retrieval systems, identities and permissions, software dependencies, non-AI system components, infrastructure, user interfaces, and human workflows. An AI red team exercise can be organized into three phases: planning and scoping the exercise, executing the selected exercises, and assessing the results to guide reporting and remediation. 1. **Plan and Scope** - Document the system's intended use, deployment environment, users, sensitive data, connected resources, and potential consequences of failure or misuse. Diagram the system's components, trust boundaries, data flows, external services, human decision points, and training- and inference-time access points. - Establish rules of engagement covering authorized systems, accounts, data, techniques, test windows, resource limits, escalation procedures, evidence handling, and stop conditions. Plan destructive, privacy-invasive, or high-cost tests for isolated environments with appropriate safeguards. - Develop a threat model based on the system's operating environment and relevant adversary behavior. Define the adversary's objectives, access, knowledge, capabilities, resources, and constraints, and account for digital and physical attack paths and the role of human oversight. - Use ATLAS tactics, techniques, and procedures to identify relevant adversary behaviors and construct threat vectors. Prioritize them according to likelihood and severity of impact to the system. - Define success criteria and stopping conditions. Identify task-level metrics for effects on the AI capability and operational metrics for measuring impact to the overall system. 2. **Execute** - Conduct the selected exercises using manual and automated methods as appropriate. Automation can generate input variations, replay attack sequences, and evaluate responses at scale. Human testers can develop system-specific attacks, adapt to observed defenses, and investigate unexpected behavior. - Follow the rules of engagement and record attack activity, system responses, control behavior, deviations from the test plan, and evidence needed to evaluate the results. - Stop or escalate testing when predefined conditions are reached. After testing, remove test accounts, modified data, installed software, persistent instructions, and other exercise artifacts. 3. **Assess, Report, and Improve** - Evaluate the results against the defined task and operational metrics. Document successful and unsuccessful attacks, their consequences, observed control behavior, deviations from the test plan, and gaps in the threat model. - Report findings to the appropriate developers, defenders, operational teams, risk owners, and other stakeholders. - Assign findings to responsible owners, track remediation, and retest corrected systems. - Use demonstrated attacks to improve preventive controls, detection, incident response, and recovery. Where appropriate, convert confirmed failures into regression tests, evaluation datasets, detection logic, monitoring requirements, or deployment criteria. Red-teaming is a continuous process and should be repeated as the threat landscape evolves and when changes are made to the system, its components, intended use, or deployment environment. New threat intelligence, vulnerabilities, and test results should inform the scope and priorities of future exercises.