Generative AI Model Alignment
Generative AI Model Alignment is the process of training or fine-tuning a generative model to guide its output toward a set of human values, goals, or ethical principles. Desired principles can be in alignment with safety, security, content policies, and legal requirements, with goals such as removing harmful content or hallucinations, reducing biases, increasing predictability, and ensuring models do not have adverse societal consequences. In the context of agentic AI, alignment is about producing models that make decisions in support of the goals of the organization and individual users, with actions that remain in scope of the designated task, and designing agents that follow security standards, exhibit transparency in decision making, and operate within organizational boundaries. Common methods for aligning a generative model during training or fine-tuning include[[ibm-llm-alignment]][[meta-llama]]: - Reinforcement Learning from Human or AI Feedback (RLHF or RLAIF) - Supervised Fine-Tuning (SFT) - Targeted Safety Context Distillation - Instruction Tuning - Direct Preference Optimization (DPO) - Constitutional AI For agentic systems, alignment depends on the purpose of the agent, the tools available to the agent, and the levels of privileges and risks associated with tool calls. Standard alignment methods for LLMs can often be used in conjunction with domain-specific fine-tuning data, reward signals with examples of correct tool calls, or reinforcement learning based on interactions with a realistic environment. Examples of alignment methods specific to agentic AI are: - Trajectory Preference Optimization[[tpmm-dpo]] - Environment-Driven Reinforcement Learning[[envrl]] - Reinforcement Learning with Execution Feedback (RLEF)[[rlef]] - Reinforcement Learning with Verifiable Rewards (RLVR)[[nvidia-mastering-agentic]] Generative AI Model Alignment should be used in combination with [Generative AI Guardrails](mitigations/AML.M0020) and [Generative AI Guidelines](mitigations/AML.M0021). Models should be continually evaluated for alignment and retrained or fine-tuned systematically to incorporate observed failure modes or updated policies. It is important to also consider that fine-tuning a model can remove previously learned alignments and should be undertaken with care and a comprehensive plan for testing the updated model for safety and security.
Confidence
Generative AI Model Alignment is the process of training or fine-tuning a generative model to guide its output toward a set of human values, goals, or ethical principles. Desired principles can be in alignment with safety, security, content policies, and legal requirements, with goals such as removing harmful content or hallucinations, reducing biases, increasing predictability, and ensuring models do not have adverse societal consequences. In the context of agentic AI, alignment is about producing models that make decisions in support of the goals of the organization and individual users, with actions that remain in scope of the designated task, and designing agents that follow security standards, exhibit transparency in decision making, and operate within organizational boundaries. Common methods for aligning a generative model during training or fine-tuning include[[ibm-llm-alignment]][[meta-llama]]: - Reinforcement Learning from Human or AI Feedback (RLHF or RLAIF) - Supervised Fine-Tuning (SFT) - Targeted Safety Context Distillation - Instruction Tuning - Direct Preference Optimization (DPO) - Constitutional AI For agentic systems, alignment depends on the purpose of the agent, the tools available to the agent, and the levels of privileges and risks associated with tool calls. Standard alignment methods for LLMs can often be used in conjunction with domain-specific fine-tuning data, reward signals with examples of correct tool calls, or reinforcement learning based on interactions with a realistic environment. Examples of alignment methods specific to agentic AI are: - Trajectory Preference Optimization[[tpmm-dpo]] - Environment-Driven Reinforcement Learning[[envrl]] - Reinforcement Learning with Execution Feedback (RLEF)[[rlef]] - Reinforcement Learning with Verifiable Rewards (RLVR)[[nvidia-mastering-agentic]] Generative AI Model Alignment should be used in combination with [Generative AI Guardrails](mitigations/AML.M0020) and [Generative AI Guidelines](mitigations/AML.M0021). Models should be continually evaluated for alignment and retrained or fine-tuned systematically to incorporate observed failure modes or updated policies. It is important to also consider that fine-tuning a model can remove previously learned alignments and should be undertaken with care and a comprehensive plan for testing the updated model for safety and security.