New Research Reveals How Large Language Models Develop Value Alignment During Training

Summary

A new research paper titled "Value Drifts: Tracing Value Alignment During LLM Post-Training" investigates how large language models learn to align with human values during the post-training phase. The study, which analyzed models including Llama-3 and Qwen-3, tracked when and how value alignment emerges through supervised fine-tuning (SFT) and preference optimization algorithms. The researchers discovered that the SFT phase is critical for establishing a model's foundational values, while subsequent preference optimization has limited ability to significantly alter these values. The research also found that different preference optimization algorithms produce varying alignment outcomes even when trained on identical preference data, suggesting that algorithm selection plays a crucial role in shaping model behavior.

• Findings provide actionable guidance for data curation and algorithm selection to improve LLM alignment with human values

Editorial Opinion

This research addresses a critical gap in LLM alignment research by moving beyond static evaluations of fully-trained models to examine the dynamic process of value learning. The finding that SFT establishes foundational values while preference optimization has limited re-alignment capacity suggests that practitioners should focus alignment efforts earlier in training rather than relying on final-stage preference optimization. The discovery that algorithm choice matters independently of data quality is particularly valuable, as it provides a new lever for improving model alignment without requiring extensive dataset curation.