NeurIPS x Appen: Evaluating Long-Horizon Agents

NeurIPS x Appen: Evaluating Long-Horizon Agents

Trajectories, verifiers and benchmark validity: a technical panel on how to tell whether an agent actually did the task, and whether the benchmark can show it.

When: Tuesday 8 December 2026, 6:00 – 9:30 PM AEDTWhere: Venue revealed on approval (5-minute walk from ICC Sydney)Registration: Registration requires approval. Approved guests will receive the venue details. Space is limited, and we prioritise researchers and practitioners attending NeurIPS 2026.

AboutAgents now run for hundreds of steps across tools, browsers and codebases, but most evaluations still check only the final state. A pass can hide a lucky or broken trajectory. A fail often can't be traced to the step that caused it. Automated verifiers and rubric-based LLM judges can score trajectories at scale, but they bring their own bias, they can be gamed, and they drift away from what a human expert would accept. Meanwhile, benchmark scores move with contamination, harness choices and run-to-run variance, so it is hard to tell whether a gap between two agents is real.

Many NeurIPS 2026 papers and workshops are working on this problem. This panel brings together researchers who build agent benchmarks, verifiers and evaluation pipelines to compare methods: what to score, who checks the checker, how much human-labelled data a judge needs, and what evidence a benchmark result should come with.

Technical discussion, not product demos.

We're your local hosts. Appen started in Sydney almost 30 years ago, and it's still home for us. We know the city well and are glad to welcome NeurIPS to it.

What we'll dig into

Outcome or trajectory: when is a final-state check enough, and when do you need step-level scoring and failure attribution?

Who verifies the verifier: how do you calibrate an LLM judge or rubric against expert humans, and how large does that calibration set need to be?

Reward hacking and lucky passes: how do you catch an agent that games the environment before the result ships?

Benchmark validity: what should an agent benchmark report (item-level results, variance across runs, contamination checks) before a score gap counts as real?

From benchmark to deployment: which evaluation signals carry over to enterprise workflows, and which don't?

Format

6:00 PM: Doors open, drinks

7:00 – 8:00 PM: Panel (45 min) and audience Q&A (15 min)

8:00 – 9:30 PM: Drinks, canapés and networking

Panelists

Moderator: to be announced

Panelists: to be announced

About AppenAppen builds human data for frontier AI, with 30 years in the industry. Today, we work with most of the world's top LLM builders. Our data products cover Frontier Alignment (reasoning traces, SME RLHF, SFT and red teaming), Agentic AI (golden trajectories, RL environment design, failure-mode taxonomy and deep evaluation), Multimodal and Speech AI, Physical AI, Model Integrity (hallucination benchmarking and bias audits) and Research Services (calibrated LLM-as-a-judge and custom benchmarking). Our off-the-shelf catalog holds 646 datasets in nine categories, including RL tasks and verifiers, each with a spec sheet covering source, consent, annotation method and known limitations.