Red teaming frontier AI at the UK AI Security Institute

Speaker: M. Pawan Kumar (AI Security Institute)
Series: CCAIM Seminar Series

Abstract: The UK’s AI Security Institute (AISI), situated within the Cabinet Office, works to equip government with an evidenced understanding of the risks posed by advanced AI. AISI's Red Team stress-tests frontier safety measures: finding ways misuse safeguards can fail, investigating unintended and harmful model behaviours, and assessing whether protocols intended to keep models under control actually work.

In this talk, I will introduce AISI and take you inside the Red Team’s research. I’ll explain how the team leverages expertise and creativity in adversarial machine learning to uncover safety failures, establish whether they reflect risks in real deployments, and understand what causes them. Red Team findings go directly to frontier AI developers to help improve their systems, and to UK and allied governments to inform decisions about AI risk and mitigations.

For students and researchers interested in frontier AI safety and security, or tackling difficult technical problems with direct public impact, the talk will offer a window into the field’s open research questions, what working on them looks like, and opportunities to join AISI.

Find out more about careers at AISI: https://aisi.gov.uk/careers

Bio: M. Pawan Kumar is a Research Director at AISI. Prior to that, he was a Research Scientist at Google DeepMind, and a faculty member in the Department of Engineering Science at the University of Oxford. Alarmed by the enormous potential harms of AI, he has recently shifted his research focus away from AI capabilities towards AI security.