Bay Area Tech Economics Seminar with Rehan Khan, Snowflake

Talk Title: Data Science at Snowflake in the Age of AI

Snowflake is a cloud data platform that thousands of enterprises use to store, analyze, and build AI applications on their data. In this talk, I describe two facets of data science at Snowflake: learning about cause and effect through experimentation and transforming how we work through the rapidly evolving capabilities created by AI. Whereas experimentation has been a staple of data science for years, in the context of relatively small sample sizes and low risk tolerance, we have needed to develop more complex regimes than simple A/B testing to guide product development and improve our systems. Our samples are counted in thousands of accounts rather than millions of users, our outcomes are heavy-tailed, and a bad treatment can disrupt customers' critical workloads. I'll describe how we still learn causally in this setting through careful choice of randomization units, switchback designs, variance reduction, and staged rollouts with guardrails. In contrast to this traditional work, there is a larger phenomenon of how AI is transforming the way our team works. In a world where teams of agents carry out data science workflows, from monitoring pipelines and investigating metric shifts to producing recurring analyses, the premium for data science work is shifting to designing robust agentic workflows, exercising judgment grounded in deep technical knowledge, and communicating meaning with clarity.

Speaker: Rehan Khan, VP of Data Science, Snowflake

The series is co-sponsored by the University of San Francisco's Master's in Applied Economics and the Stanford Causal Science Center. Join the seminars mailing list!