Testing the Untestable: Reliably testing AI-Infused Applications

Testing is hard, which is why developers tend to avoid it. Testing non-deterministic things is even harder—an unfortunate reality now that you are likely building AI-infused applications where unpredictability is the norm.

What happens when your applications start using advanced features, such as RAG, tools, and agents? How do you ensure reliability without blowing your budget?

In this session, join Eric for a practical, code-focused tour through the specific tips, technologies, and practices designed to solve these problems. Through concrete examples and demonstrations, you will explore testing strategies across three critical levels:

1. Unit Testing: How to isolate and verify your applications at the unit level.
2. Integration Testing: How to ensure your application behaves once wired up to “real” models.
3. End-to-End Testing: How to validate things still work as expected after making changes.

The AI landscape changes at a breathtaking pace. This session will help you cut through the noise so that, by the end, you will be able to confidently answer the ultimate question: "If I change my model, prompt, or application, did I get better or worse?"

Note: This is a hybrid online/in-person event. Food and social for those in the OCI training room starts at 6:00pm. Meeting starts at 6:30pm for those online and in the training room. The speaker will be remote.