Simulated-User Evaluation Infrastructure: Evaluating AI Products with 8.3 Billion Persona Agents

Caltech World Model Club × Caltech C Seminar Series: Simulated-User Evaluation Infrastructure: Evaluating AI Products with 8.3 Billion Persona Agents

We are excited to welcome Haixin Wang, Founding Engineer of MatrAIx, for the next talk in our joint seminar series! MatrAIx is building simulated-user evaluation infrastructure powered by 8.3 billion persona agents. Its research brings together leading AI researchers from Harvard, MIT, Stanford, and Berkeley, with an advisory committee featuring Nobel Laureate Vernon Smith, Microsoft Chief Scientific Officer Eric Horvitz, and MIT CSAIL Director Daniela Rus.

Learn more about MatrAIx: https://matraix.ai/

RSVP if you'd like to learn how world models can simulate human behavior and transform AI product evaluation, or if you're just curious. Open to the LA community!

Event Details

Speaker: Haixin Wang, Founding Engineer of MatrAIxDate: Friday, October 30, 2026Time: 5:30–6:30 PMLocation: Annenberg 105, Caltech (🍕 free food provided)

Abstract

AI products are usually judged on offline benchmarks, but a model that ranks first on a leaderboard can still disappoint the people who actually use it. In this talk, I will introduce MatrAIx, a simulated-user evaluation infrastructure built on a population of 8.3 billion persona agents. Each persona is described with a shared schema of 1,290 attributes covering background, psychology, capability, behavior and lifestyle, and the population combines personas extracted from real-world sources with synthetic generation. I will show how we sample persona cohorts to evaluate four kinds of products (surveys, chatbots, web products and apps) across more than 1,000 tasks in 25+ domains, and how the results show which users a product works for and which it fails. I will also cover how we check that persona agents behave as specified, and where we see simulated users going next, from personalized AI agents to embodied persona AI.