Voice Your World: An LLM Pipeline for Speech-Driven Interaction with Virtual Environments

What if you could simply tell a virtual world what to do? This project explores voice as a playful way to interact with a 3D environment on the Meta Quest, replacing menus and complex interfaces with a few simple controls and natural speech. Built in Unity, the pipeline uses Whisper to turn speech into text, then passes the transcript to a locally hosted large language model. The model translates free-form requests into structured commands, letting users shape the world simply by describing what they want. The project is an ongoing experiment, and this talk shares the architecture, what works so far, and the surprises found along the way.

The talk will start at 19.30.Pizza & Beers are on us! 🍕🍻