CS Seminar Series: Ruofei Du

Kicking Off the Flywheel of AI + XR Innovations with XR Blocks

Abstract:

The concurrent revolutions in generative AI and spatial computing are fundamentally transforming how we work, communicate, and live. While AI research is accelerated by mature frameworks and benchmarks, inventing AI-driven XR interactions remains a high-friction process. This raises a critical question: how can we make AI + XR technology more accessible and useful to creators?
This talk first explores a continuum of research designed to augment human abilities in XR, including Geollery, Depth Lab, Visual Captions, Sensible Agent, and Agent Hands. Building upon foundational systems like Visual Blocks and InstructPipe, we empower creators with visual programming tools that better align with human intent.
Finally, we present XR Blocks (https://xrblocks.github.io), a cross-platform framework to accelerate human-centered AI + XR innovation. Driven by the mission of “minimizing code from idea to reality,” XR Blocks provides core abstractions and samples that empower creators to move from concept to interactive AI + XR applications. We conclude the talk with live demonstrations of Vibe Coding XR (https://xrblocks.github.io/gem), offering a visionary glimpse into the future and the automatic evolution of the AI + XR flywheel.

Bio:

Ruofei Du serves as the Interactive Perception & Graphics Lead at Google XR, where he drives AI + XR innovations. An established expert in his field, he served as an Associate Chair on the program committees for both ACM CHI (2021-2026) and UIST (2022-2026), and an Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology. Dr. Du's contributions includes 8 US patents and over 45 peer-reviewed publications in top venues across HCI, Computer Graphics, and Computer Vision. His research has garnered significant recognition, including the Auggie Award at AWE USA 2026, a Distinguished Paper Award in ACM IMWUT, Best Paper Award at SIGGRAPH Web3D 2016, and multiple Honorable Mentions Awards at CHI and TVCG. He holds a Ph.D. in Computer Science from the University of Maryland, College Park. Website: https://duruofei.com