Computing Colloquium featuring Dr. Rory Barnes

Computing PhD Colloquium

Title: Integrating AI Coding Agents into Scientific Research

Presented by: Rory Barnes, Ph.D., Astronomy, University of Washington - Professor of Astronomy, Astrobiology, Data Science

Abstract:

New software tools have emerged that enable AI agents to quickly develop code that analyzes data and generates figures, but their mistakes can also harm a researcher's computer and lead to enhanced skepticism of AI-assisted results. These concerns could be mitigated if scientific software is developed in a secure environment with a system for monitoring changes in files and results. I explore these issues through the lenses of reproducibility standards, software engineering best practices, and artifact custody methods. I first identify a dozen ways in which AI agents change how scientific software and its results are generated, verified, and reproduced. I then propose a six-level scale that classifies AI-assisted numerical experiments by their provenance, reproducibility, openness, oversight, and falsifiability (PROOF). The scale begins with a locally self-consistent version-controlled code repository and culminates in a publicly accessible and curated archive of the files, change history, prompt transcripts, independent agent regeneration of the analysis, and multiple cryptographically verifiable attestations that the results have not been corrupted by AI agents. To help facilitate this safe and reproducible development of numerical experiments with AI agents, I created an open-source software package called vaibify that builds secure environments connected to a graphical interface where scientists can prompt command line agents, view the current status of the local and remote files, display figures, and be guided up the first three rungs of the PROOF Ladder. While the veracity of scientific results remains the domain of humans (for now), vaibify can at least ensure that the digital artifacts were generated in the correct order, local files match remote files, and the results are reproducible at the bit level over decade-long timescales.

Speaker Bio:

Rory Barnes is a professor of Astronomy, Astrobiology, and Data Science at the University of Washington in Seattle. His research focuses on developing computational models of planetary system evolution, with a focus on planetary habitability. He has also developed software packages that leverage machine learning to infer Bayesian uncertainties in high-dimensional problems. Lately, he has turned his attention toward exploiting AI coding agents to improve the quality of scientific software. Prof. Barnes began his career at the University of Arizona and received his Ph.D. in astronomy from the University of Washington, joining the faculty in 2013. Rory Barnes is a professor of Astronomy, Astrobiology, and Data Science at the University of Washington in Seattle. His research focuses on developing computational models of planetary system evolution, with a focus on planetary habitability. He has also developed software packages that leverage machine learning to infer Bayesian uncertainties in high-dimensional problems. Lately, he has turned his attention toward exploiting AI coding agents to improve the quality of scientific software. Prof. Barnes began his career at the University of Arizona and received his Ph.D. in astronomy from the University of Washington, joining the faculty in 2013.