Generative AI is reshaping personalization by making user experiences more expressive, intent-driven, and increasingly defined in natural language. Outputs become open-ended, contextual, and fine-grained, which disrupts much of the evaluation infrastructure that large-scale personalization has historically relied on. In these settings, “ground truth” is often underspecified, and quality depends on criteria like relevance, coherence, and user alignment rather than exact-match correctness. Yet human review and user studies do not scale. This talk explores LLM-as-a-Judge as a practical evaluation layer for generative and agentic systems. Drawing on recent research and public case studies, including published work from Spotify, we discuss how LLM-based judges can approximate human judgments, connect offline evaluation with human perception, and support meaningful assessment in low-feedback and cold-start settings. We then move from evaluation to optimization. When the judge signal is reliable, it can serve as an alignment signal for post-training and as a driver for automatic prompt and context optimization, enabling systematic comparisons, regression detection, and faster iteration cycles toward production-ready behavior.
