Stanford CS329A Part 8 — Agentic Evaluations and Long-Horizon Tasks

How capable is an AI agent? Stanford CS329A Part 8 shows why one leaderboard cannot answer that question. Human-time horizons, professional deliverable quality, and evidence-grounded research synthesis expose different strengths and different failure modes.

This teaching companion uses the complete official-video transcript, 18 inspected visual cues, and pinned primary-paper revisions. The DeepScholar-Bench figures are versioned explicitly because the Autumn 2025 lecture and the current 2026 paper revision report different aggregates.

Download the Slides

The Core Idea

                 agent capability claim
                         |
       +-----------------+-----------------+
       |                 |                 |
       v                 v                 v
  task duration     deliverable value    evidence quality
       |                 |                 |
       +-----------------+-----------------+
                         |
                 deployment decision

Each branch has a different unit, protocol, and blind spot. Combining them is a teaching recommendation, not a metric proposed by one paper.

1. Time Horizons Measure a Reliability Boundary

The METR work starts with tasks timed on skilled humans. Agent success is measured across those tasks, then fit against human completion time.

agent success
100% |\
     | \
 50% |--X  <- 50% horizon
     |    \
  0% +-----\------------ human completion time

The 50% time horizon is the fitted duration where model success reaches 50%. It is not the longest task an agent completed once.

The choice of threshold matters:

METR v4, revised 2026-07-10, reports a Claude 3.7 Sonnet 50% horizon around 50 minutes and an approximate seven-month historical doubling trend since 2019. The paper explicitly discusses external-validity limits. Its fitted trend must not be treated as an unconditional forecast of workplace automation.

2. Long Tasks Fail Through Trajectories

The lecture’s failure analysis includes:

• poor planning or tool choice;
• incorrect reasoning or mental models;
• premature task abandonment;
• repeated failed actions.

These failures compound over time. A final pass/fail score does not show whether the system chose the wrong tool early, entered a loop, or abandoned a recoverable path.

Human time is also an imperfect difficulty proxy. Estimates depend on expertise, familiarity, tools, instructions, and repository context.

3. GDPval Measures Professional Deliverables

GDPval changes the target from duration to economically valuable output.

professional task brief + reference context
                    |
              model deliverable
                    |
       pairwise expert comparison
                    |
            preference judgment

The benchmark covers 44 occupations across nine high-GDP U.S. sectors. The lecture describes 1,320 tasks and a 220-task open gold subset; the paper releases the 220-task subset.

The paper reports roughly linear improvement across its compared model snapshot and says leading systems approach experts in aggregate deliverable quality. That does not mean:

• parity on every task or occupation;
• autonomous discovery and execution of an entire workflow;
• job replacement;
• equal performance without supplied context or scaffolding.

Performance varies with occupation, modality, task duration, context, reasoning effort, and scaffolding. Aggregate near-parity can hide important weak areas.

4. Research Synthesis Needs Three Scores

DeepScholar-Bench evaluates a live related-work task.

An oracle-source experiment can provide the target references. If performance remains weak, the bottleneck is no longer only retrieval; evidence extraction and synthesis also matter.

5. The DeepScholar Version Boundary

This discrepancy must remain visible:

Do not subtract or connect these numbers. The paper revision, evaluated systems, or aggregation details may differ. They are not directly comparable, are not a time trend, and do not establish a 12-point improvement.

The stable conclusion across both snapshots is narrower: the benchmark remains far from saturated, and no system simultaneously solves synthesis, retrieval, and verifiability.

6. Build an Evaluation Portfolio

For a deployment claim, record the protocol before the score:

Long duration does not guarantee useful output. Useful deliverables do not establish autonomous work. Fluent research prose does not guarantee source coverage or support.

The Lecture in 12 Teaching Moves

Homework: Evaluate a University Research Agent

Design three tasks for an AI research assistant:

1. a multi-step task with a skilled-human time estimate;
2. an economically or academically valuable deliverable judged pairwise;
3. a cited related-work synthesis graded for synthesis, retrieval, and verifiability.

For each task, state the threshold, supplied context, judge, expected failure mode, and blind spot. Conclude what evidence would be necessary before allowing unattended use.

Source Versus Teaching Synthesis

Directly sourced:

• METR’s human-calibrated time-horizon method, historical fit, and failure analysis;
• GDPval’s occupational coverage, task construction, pairwise expert evaluation, and aggregate findings;
• DeepScholar-Bench’s three dimensions and version-specific aggregate statements;
• the lecture’s cross-benchmark discussion.

Teaching synthesis:

• the three-axis evaluation portfolio;
• the deployment audit table;
• the university research-agent homework;
• the rule that every slide should expose distribution, threshold, context, version, and blind spot.

References

1. Stanford CS329A course site.
2. Official Part 8 video.
3. Thomas Kwa et al., Measuring AI Ability to Complete Long Software Tasks, v4, 2026.
4. Tejal Patwardhan et al., GDPval, v1, 2025.
5. Liana Patel et al., DeepScholar-Bench, v2, 2026.

Related Reading

• Stanford CS329A — Self-Improving AI Agents: Public Lecture Learning Path
• Stanford CS329A Part 7 — Self-Improvement and Deep Research Agents
• Stanford CS329A Part 9 — Future Research Areas