PalanerFree on the App Store
Get
Evaluation of AI Agents: Quality vs Cost
WorkFree

Evaluation of AI Agents: Quality vs Cost

Tuesday 17 November 2026, 14:30Venue time (Warsaw)
Faculty of Physics, University of WarsawDirections
FreeTickets sold by luma.com
Book · Freevia luma.com
Add to calendar

Downloads an .ics file · Times are in Europe/Warsaw · Google Calendar

We can use various agents: Claude Code, Codex, Aider with DeepSeek, pi.dev, Mirror with Beam and others, but do we choose them based on their real capabilities or marketing? Let's check in practice: give them the same tasks and see what happens.

We will perform tasks from simple to tricky. Each agent works alone, then we compare results.

What we look at

Whether the solution works and handles edge cases

How fast and efficient the generated code was

Token usage and their cost

How many times a human had to step in and fix the direction

The same model in a different tool can perform differently. We want to see how the harness changes the model's results on the same task.

Evening plan

Quick intros, who starts with what.

Rounds from simple to tricky, with various challenges.

Results board.

Discussion, why one succeeded and another didn't; what stacks, prompts, configurations we have.

Technical requirements: Linux or macOS. It's good if you have your own tokens for the agent; if not, please inform before the event.

For whom: for everyone who codes with AI and wants to see how different setups perform on real tasks. Have an idea for a tricky task? We'll be able to run and test it if we have time.

Ask Palaner

Going to Evaluation of AI Agents? Ask me anything about it.

I read the organiser's pages and answer in a few seconds.

Answers are AI-generated · Privacy