Interactive learning lab
Build a Verifiable Agent Loop - Six-Question Learning Lab
Learn the improvement-loop model, then work through the same six demanding questions that developed it.
Progress stays in this browser.
Teach first
Improve with evidence, not confidence
First learn the system. Then answer six questions without seeing the learner case or enhanced answer in advance.
Mental model
A response is not an improvement system
Model responserequest → model → answer
Agent loopcontract → attempt → evidence → targeted repair → stop
Suitability gate
Four questions before automation
- Specifiable?
- Can you state the desired result and constraints?
- Verifiable?
- Can evidence check quality independently of confidence?
- Repairable?
- Can a detected defect be changed and rolled back?
- Bounded?
- Can retries, cost, access, and authority be limited?
Worked example
One lecture becomes ten slides
The agent may help create the deck, but it cannot certify its own interpretation.
- Contract: ten slides, one lecture, source fidelity, readable rendering, two revisions.
- Attempt: create a source map, teaching sequence, and first rendered deck.
- Verify: check timestamps, claims, links, dimensions, overflow, and visual hierarchy.
- Diagnose: identify the slide, defect, and evidence rather than saying "improve it."
- Revise: change only failed slides and rerun affected checks.
- Stop: pass the contract, exhaust the budget, or hand ambiguity to the teacher.
- Persist: save a verified rule or test, then evaluate it on a different lecture.
A saved rule, checklist, test, tool, or governed memory must improve performance on a future task. Human approval of one correction is not enough.
Open the short glossary
- Contract
- The goal, evidence, constraints, permissions, budget, and handoff rule defined before generation.
- Verifier
- A check that can expose a specific defect. It supplies evidence, not automatic truth.
- Diagnosis
- A location-specific explanation connecting a failed check to evidence.
- Candidate coverage
- Whether any generated candidate contains a correct or useful answer.
- Selection reliability
- Whether the system identifies and returns the better candidate.
- Persistence
- What changes after the run: a rule, test, tool, memory, artifact, data, or model behavior.
This static page does not intentionally upload your answers. Responses are stored only in this browser's local storage. Do not enter private, confidential, or identifying information.
Challenge 1 of 6 About 3 minutes
Reconstruct the loop
Without checking the teaching section, reconstruct the agent improvement loop using approximately seven verbs.
Define → ______ → ______ → ______ → ______ → ______ → ______
Optional hint
The loop needs generation, evidence, a reason for changing something, and a terminal decision.
Self-check before comparison
Real first-pass learner case
"Plan, review, implement, audit, improve, loop."
What it recognized: work should be planned, checked, changed, and repeated.
What remained weak: review and audit overlap; the answer lacks an explicit contract, evidence-linked diagnosis, and stop or handoff condition. A loop is the bounded return path, not the final stage.
Contract → Attempt → Verify → Diagnose → Revise → Reverify → Stop or hand off.
Challenge 2 of 6 About 4 minutes
Write the contract before generation
Suppose an agent must create a ten-slide lesson from one Stanford lecture. What should the contract specify?
- Goal
- Required evidence
- Prohibited behavior
- Retry or human-approval rule
Optional hint
Watching a video is a process. A claim-to-timestamp map is inspectable evidence.
Self-check before comparison
Real first-pass learner case
A useful handoff rule, but process and evidence were mixed
"Create a ten-slide deck for the selected online lecture. Watch the lecture locally, check the transcript and screenshots, and arrange the topics consistently. Do not skip watching the videos. If the video or transcript cannot be accessed, ask for approval and suggest an operation."
What it recognized: source access matters, skipping source review is prohibited, and uncertainty should trigger human approval.
What remained weak: "selected lecture" conflicts with "all videos"; watching is not evidence of accurate interpretation; readability, claim support, revision budget, and final authority are unspecified.
Create ten slides that accurately teach one selected lecture. Map every major claim to a timestamp or approved source; render and inspect all slides. Do not invent claims, quotations, timestamps, or inaccessible-source coverage. Allow two revision cycles. Stop with the exact evidence gap and options when access or interpretation remains uncertain; the student or teacher approves publication.
Challenge 3 of 6 About 5 minutes
Design four verifiers
The agent produced ten slides. Define one source-fidelity check, one content-completeness check, one visual-quality check, and one teaching-quality check.
Each check must produce PASS, FAIL, or NEEDS HUMAN JUDGMENT.
Optional hint
Separate machine-checkable defects such as broken links or overflow from interpretive judgments such as whether the teaching narrative is appropriate.
Self-check before comparison
Real first-pass learner case
Multiple reviewers were proposed, but most verdicts remained subjective
"Check whether slide content appears in the transcript or screenshots; cover the video from beginning to end; ask a simulated student whether slides are clear; ask teacher and student subagents to review, and send disagreements to a human."
What it recognized: source support, coverage, readability, multiple perspectives, and human escalation all matter.
What remained weak: screenshots rarely verify factual claims; chronological coverage can become transcript compression; simulated students do not prove readability; agreement between agents is not authority; "okay" has no observable threshold.
- Source: PASS when every major claim maps to a timestamp or approved source; FAIL for unsupported or overstated claims.
- Completeness: PASS when the central question, mechanisms, examples, limitations, and conclusion are represented; HUMAN for the best teaching narrative.
- Visual: PASS when renders show no overflow, collision, cutoff, broken media, or unreadably small text; HUMAN for aesthetics and pacing.
- Teaching: reviewer agents apply a defined rubric for objectives, definitions, examples, and misconceptions; their feedback informs diagnosis, while a teacher judges educational appropriateness.
Challenge 4 of 6 About 4 minutes
Diagnose an overclaim
Slide 6 says: "Generating more answers makes an agent more reliable."
The transcript says that more candidates increase the chance that one is correct, but the verifier may still select the wrong candidate.
- What exactly is defective?
- What evidence identifies the defect?
- What targeted revision should be made?
- What should be rechecked afterward?
Optional hint
Candidate coverage and selected-answer reliability are different measurements.
Self-check before comparison
Real first-pass learner case
The central overclaim was found, but the diagnosis added unsupported reasoning
"The slide claim is too strong: the transcript only says chance, but the slide says it will. More answers may create contradictions and confuse the agent. Revise it to say more answers increase the chance that one candidate is correct but need a human to verify. Then run it multiple times and compare."
What it recognized: the slide strengthened a probabilistic statement into a guarantee.
What remained weak: contradiction and confusion were not evidence supplied by the transcript; humans are only one possible verifier; generating more runs does not independently identify the correct candidate.
The defect conflates candidate coverage with returned-answer reliability. Revise to: "Generating more candidates can improve coverage, but final-answer reliability improves only when the verifier can identify the better candidate." Recheck the timestamped source, adjacent slides for the same overclaim, and - if evaluating the system - measure candidate coverage and selected-answer accuracy separately.
Challenge 5 of 6 About 3 minutes
Stop without hiding uncertainty
After two revision cycles, source and rendering checks pass. The teacher reviewer approves, but the student reviewer says Slide 8 remains too technical. The retry budget is exhausted.
- Final status
- What the agent must not do
- What evidence and unresolved issue it gives the human
Optional hint
Stopping does not require pretending that the artifact is fully complete or completely failed.
Self-check before comparison
Real first-pass learner case
The learner correctly chose to stop and preserve the disagreement
"Stop, save the current progress and review, diagnose the issue on Slide 8, and ask for human verification with suggestions. Do not ignore the student reviewer, declare the job done, or hide the Slide 8 feedback."
What it recognized: budget exhaustion requires stopping, not silent extra work; disagreement and evidence must survive the handoff.
What needed precision: label the status NEEDS HUMAN JUDGMENT; diagnosis can continue, but revision needs authorization. Provide concrete options rather than a vague request.
Status: NEEDS HUMAN JUDGMENT. Hand off the current deck, revision history, passed source/render checks, exact Slide 8 concern, diagnosis, and exhausted budget. Offer options: approve for an advanced audience, simplify terminology, add an example, split the slide, or authorize one more cycle. Do not declare success, suppress feedback, revise beyond the budget, or self-authorize publication.
Challenge 6 of 6 About 4 minutes
Make the lesson persist
The agent repeatedly converted cautious source language such as "may improve" into strong claims such as "will improve." What should persist so the next project benefits?
- One artifact to save
- One new verification rule
- One thing that should not be stored
- How to test whether the next run improved
Optional hint
Rerunning the same example tests regression. A different lecture tests whether the lesson generalizes.
Self-check before comparison
Real first-pass learner case
A reusable skill was proposed, but evaluation needed a held-out task
"Create a skill that records and analyzes similar mistakes. Check every strong claim against the source language. Do not store the actual source content. Run the task again and compare the result using the new skill or rule."
What it recognized: turn the failure into a reusable procedure, calibrate claim strength against the source, avoid unnecessary source retention, and compare performance.
What needed precision: one example can overfit the skill; full source content is unnecessary and may create permission or privacy problems; rerunning only the original task tests regression, not generalization.
Save a claim-calibration checklist or skill containing the generalized failure pattern, strength taxonomy, verification procedure, and permitted minimal examples. For every major claim, locate the source, classify it as direct statement, paraphrase, or inference, and flag unjustified shifts from "may" to "will." Do not retain complete transcripts or unauthorized material when pointers and minimal examples suffice. Test the original lecture for regression and a held-out lecture for generalization; compare overstated claims, unsupported claims, false warnings, human corrections, time, and cost under the same conditions.
Your learning artifact
Agent-loop playbook
This playbook uses your final answers when available and your first answers otherwise. Edit it before exporting.
Copied, printed, or downloaded reflections leave this page's local storage. You are responsible for managing the exported file.