Measuring Codelab Success with AI-Simulated Developer Personas
Running one Google Codelab twice — as a newcomer and as an expert — with a frontier model and parallel sub-agents doing the work
Disclaimer. This study is itself an experiment: using AI to evaluate a developer resource, with the AI doing the work at every level — running the codelab, analysing the results, and drafting both this article and the accompanying deck. Every command, timestamp, and error message below is real, reproducible, and published alongside the raw logs. Read what follows as a documented starting point that invites further testing — and with a healthy grain of salt.
Overview
We took Google's own codelab, Deploy a website with Cloud Run (nine steps, advertised at 39 minutes) and ran it end to end twice against production Google Cloud, from two deliberately different developer perspectives. Every command was run exactly as printed, in three fresh, isolated projects. We logged wall-clock timestamps at every step boundary, and we built a concept inventory from the codelab text before either run began.
Both personas finished all nine steps. The newcomer took 30m 26s, which lands inside the advertised 39 minutes, and we're not going to build an argument on a nine-minute gap. The clock is the least interesting thing about how they got there.
What the clock hides is that the newcomer did not get there unaided. It hit PERMISSION_DENIED at step 4, followed the codelab's own escape routes to a dead end, declared itself blocked, and needed one expert intervention to continue. The experienced developer hit the same wall, at the same step, for the same underlying reason — recovered in seconds, and drew the wrong conclusion about why.
The error message is what connects the two. It prescribes the wrong fix to both of them. And the newcomer, working in the environment the codelab tells you to use, got the less informative version of it.
How we did it?
The study was orchestrated by a frontier model — Claude Opus 5 — acting as the analyst and coordinator, with sub-agents running underneath it in their own separate contexts and in parallel with the main thread.
Why Opus 5?
At the time of this writing (July 2026), Claude Opus 5 tops the Humanity's Last Exam leaderboard, and the wider Claude model family — Opus 5, Fable 5, Mythos 5, Sonnet 5 — occupies the top of the field across several dimensions, including computer use and agentic coding. Our focus for this first run was not to compare models against each other but to establish what's achievable at the ceiling of what's available today: we picked the strongest model we could get our hands on and asked it to run the entire study — execution, analysis, and writeup — unaided, against a real codelab and a real cloud account. We plan to expand this into a cross-model study, running the same two personas against other frontier models. But for this first experiment, the question was simply what the best case looks like at today's highest levels of model performance.
Best Overall (Humanity's Last Exam) rankings, screenshotted July 2026. Source: Vellum AI LLM Leaderboard.
Leaderboard sources: Vellum AI LLM Leaderboard, LMArena.
That parallelism is what made a controlled two-persona design possible at all. The two personas had to run simultaneously and in isolation — if one enabled an API before the other reached that step, the comparison would silently corrupt — so isolation was enforced by project (cl-dx-a-0726, cl-dx-b-0726). Sub-agents also carried the bounded mechanical work: capture, annotation, scoring, and PDF/deck generation, each with a file deliverable and a fast verification step.
Alongside that, the model drove the browser directly using computer use to navigate the codelab, drive Cloud Shell, take screenshots at each step, read the literal error text off the screen, and iterate. Two notes on that, because they're part of the findings:
- Computer use was how we found the study's own worst bug. Browser automation drives a Chrome tab without visually selecting it, while macOS
screencapturephotographs whichever tab is selected — so 10 of Persona A's 11 screenshots came back as stale frames of the codelab page. We caught it by hashing the files, not by looking at them. All 10 were discarded, retained for audit, and documented in the asset index; the published figures were re-captured with headless Chrome. The capture script now hash-checks itself and fails loudly. - One sub-agent overruled its coordinator. The first quarantine sweep was too broad and wrongly condemned a genuine screenshot. The Persona A sub-agent pushed back, the file was re-examined, and it was restored. One of our two published Persona A captures exists because of that disagreement.
The two personas
| Persona A — new to Google Cloud | Persona B — experienced | |
|---|---|---|
| Who | Competent developer, other stacks, no GCP background. Executed by a separate, smaller sub-agent (Sonnet) in its own context. | Experienced GCP developer with full documentation access. Executed firsthand by the analyst. |
| Environment | Cloud Shell, exactly as the codelab instructs | Local gcloud — because that's what an experienced developer actually does |
| Rules | A written clean-room protocol: act only on what the page says; escape a failure only by re-reading the step, reading the literal error, and searching the exact error string. Nothing else — no migration guides, no release notes, no reference docs, because a newcomer doesn't know they exist. Log every ambiguity before resolving it. Hard cap ~10 minutes per blocker, then declare BLOCKED and request help. | Ran every command as printed first, then recovered using expert knowledge — so both personas are measured against identical starting commands. Divergence from the written instructions is itself evidence. |
| Result | 9/9 steps · 30m 26s · 1 block, needed outside help · 56 concepts | 9/9 steps · 6m 04s · 4 blocks, all self-recovered · ~5 concepts |
Every result was plotted on two axes — measured seconds against counted new concepts — so the interesting cases separate by a counting rule rather than by anyone's judgment: the grind (trivial to grasp, burns minutes) and the trap (fast to type, completely opaque).
Eighteen cells, nine steps × two personas. Two structural results fall out of the picture: every Persona B cell is low-load — the codelab asks an experienced developer to understand nothing they don't already understand, so their entire cost is time — and Persona A has no clean cell after step 1. The sharpest cell is step 1 itself, the "Before you begin" page that asks the reader to do nothing at all and delivers 12 undefined concepts. It's cleared in 25 seconds and the vocabulary debt is carried into everything that follows.
We also recorded what the study got wrong: a pre-registered hypothesis failed, and our first conclusion about step 4 — "the codelab omits an IAM step" — was nearly published before a controlled probe overturned it. The real cause was IAM propagation delay, and the error message actively prescribes the wrong fix.
The findings
→ Read the full report — executive summary, method and limits, the two-axis result, and nine recommendations (five of which are trivial edits to the codelab text).
Also available: the 18-slide deck, the propagation probe if you want to check the central claim, and the asset index if you want to check the visuals.