Techniques for more realistic model evaluations

This week, Axel Ahlqvist will share results from his recent research paper on LLM audit realism, completed at Meridian Cambridge. As frontier models become ever more capable, they also become better at detecting when they're in a test environment. At that point, they might choose to behave in a way they otherwise wouldn't, so the results of the test can't be trusted. Therefore, a crucial aspect to building a solid understanding of what the latest models are capable of is creating environments that models can't distinguish from real deployment. Axel will present the methods they found to significantly increase test realism, followed by a group discussion. Paper preprint: Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds 📅 September 30th, 18:00 📍 Sveavägen 76 (EA Sweden Office) 🍌 Snacks will be provided Optional Reading Automated auditing (Petri): alignment.anthropic.com/2025/petri/ Paper X thread: x.com/axelahlqvist/status/2095518876268438009 Less wrong blog post: lesswrong.com/posts/9DyiNexLoyJqwNWFn/improving-audit-realism-with-inference-time-compute-and Paper preprint: arxiv.org/abs/2609.02302 On eval awareness: anthropic.com/engineering/eval-awareness-browsecomp We look forward to seeing you there! (Note that we also post our events on Facebook, so the Meetup attendee list is not indicative of the total expected number of participants.) Ring "Effektiv Altruism" when you arrive and we'll let you in, then we're up two flights of stairs.