Disclaimer. This study is itself an experiment: using AI to evaluate a developer
resource, with the AI doing the work at every level — running the codelab, analysing the results,
and drafting both this deck and the accompanying article. Every command, timestamp, and error message
here is real, reproducible, and published alongside the raw logs. Read what follows as a documented
starting point that invites further testing — and with a healthy grain of salt.
Both personas completed all nine steps. Neither completed them the way the codelab describes.
Two developers, same nine steps.
Both are competent. The difference is not ability — it is how much of Google Cloud
they already carry in their head before step 1.
Sam
Persona A · new to Google Cloud
6 years shipping product in Node and TypeScript
Deploys on Vercel and Render; knows Docker through docker-compose
Has never opened the Google Cloud console
Found this codelab searching “deploy a container to Google Cloud”
Mental model “A deploy is a git push.”
Wants something live, and to know what it will cost.
How they ran it Cloud Shell, exactly as the page instructs.
Alex
Persona B · experienced with Google Cloud
9 years in platform and infrastructure
Runs Cloud Run and GKE in production; fluent in IAM, Cloud Build, Artifact Registry
Works locally; avoids Cloud Shell by preference
Opened this codelab to decide whether to hand it to new hires
Mental model Already holds every primitive the codelab introduces.
Wants to find where it breaks before their team does.
How they ran it local gcloud — the way they actually work.
These are analytical constructs used to structure the run, not researched user archetypes.
Persona A was executed by a separate, smaller model under a written clean-room protocol; Persona B by the
analyst firsthand. Both ran in separate projects, so neither could clear a step for the other.
Naming them makes the divergence concrete. The protocol, not the persona fiction, is what makes the measurement valid.
The codelab is accurate about everything except what happens when something goes wrong.
And something goes wrong for both personas — at the same step, for the same reason,
with two different error messages. The newcomer got the less informative one.
1
A new project cannot reliably run step 4.
The error names a permission the account already has.
2
The cleanup step reports success while
leaving 356 MB billing in Artifact Registry.
3
The codelab is written for Cloud Shell and
silently punishes anyone working locally.
Point 3 is the one that generalises: the person with least context received least information.
Both finished. One took five times longer and absorbed eleven times more.
Persona A
Persona B
ratio
Time to complete
30m 26s
6m 04s
5.0×
Steps completed
9 / 9
9 / 9
—
Blocks hit
1
4
—
Needed outside help
yes
no
—
Concepts to absorb
56
5
11.2×
Why did Persona B hit four times more blocks and still finish five times faster?
1
Every block was already in their head. A hardcoded Cloud Shell
path, an empty environment variable, a width-dependent awk — Alex has debugged all three
shapes before. Recognition, not discovery.
2
They never had to leave the terminal. No searching, no reading,
no deciding whether the tutorial or their setup was wrong. Measured recoveries:
19s, 34s, 43s.
3
A block only stops you if you can't name it. Sam hit
one block and stopped dead, because the error named a permission they had no way to evaluate.
Block count measures how often the codelab breaks. It does not measure difficulty.
What separates these two columns is not how often they got stuck — it is whether being stuck meant anything.
A's raw wall clock was 40m07s; ~10 min of that was a defect in this study's own capture tooling and is excluded. Documented in scoring.md.
Persona B never has to understand anything. Persona A never gets a clean step.
Every concept this codelab introduces — revisions, concurrency, image tags, regions,
service accounts — Alex already holds. So each one costs them zero: their whole cost is waiting
for builds. Sam meets all 56 for the first time, which is why their points climb the vertical axis while
Alex's stay pinned to the floor.
Persona APersona Bnumbers are codelab steps · thresholds: 120s, 4 concepts
Every B point sits on the low-load floor. B's entire cost is time. A has no clean cell after step 1.
The friendliest page is the steepest cliff.
Step 1 asks the reader to do nothing at all. It is cleared in
25 seconds — and it delivers 12 undefined concepts,
the largest single block in the codelab.
Knative. Cloud Build. Container Registry. Cloud Shell. gcr.io. Scale-to-zero.
Containerization. Project-owner role. The stated prerequisite is
“general familiarity with Docker” and nothing else.
Persona A carries all twelve into every step that follows.
Persona B reads it in 12 seconds and holds none of it as new.
Step 1 as captured, 26 July 2026. Zero actions requested; twelve concepts introduced.
This is the purest 'trap' cell in the study: low time, highest load.
The codelab asks for one thing and assumes fifty-one more.
What the codelab says you need
“General familiarity with Docker.”
That is the complete prerequisites section. Nothing else is asked for.
What it actually uses without explaining
KnativeCloud BuildContainer Registrygcr.ioArtifact RegistryCloud Shellproject IDbilling accountregionzoneimage tagrevisionconcurrencyservice accountIAM rolescale to zerounauthenticated invocationsCloud Storage bucketenabling an APIPodsservice URL+31 more
51 of 56 concepts are used with no explanation. One is actually defined
(project ID). Four are partly explained.
A codelab may fairly assume prerequisites — but only the ones it declares. Under its own
stated contract every term above is in scope. For Sam each is a lookup or a guess;
for Alex each is free. Counted from the page before either run,
so the count cannot have been shaped by knowing what broke. A hyperlink to another doc was not counted
as an explanation.
Reframed from a percentage to the actual gap: one stated prerequisite against a wall of assumed vocabulary.
Both personas failed the same command. Only one was told why.
PERSONA A · Cloud Shell
transcript replay — not a screen recording
0:02no permission named
0:02no resource named
0:03no troubleshooter link
1:48declared BLOCKED, stopped
PERSONA B · local gcloud
transcript replay — not a screen recording
0:02names the exact permission
0:02names the principal
0:03names the resource + link
1:44self-recovered
Identical underlying condition. The newcomer, in the environment the codelab
prescribes, received strictly less information than the expert who needed it least.
This asymmetry inverts the relationship between need and information. Recommendation R2.
The obvious conclusion was wrong. A controlled probe caught it.
“The codelab omits an IAM step” is what this study nearly published.
The tell: the failing service account already held roles/editor,
which already grants the permission the error named.
18:52:51service account still holds exactly roles/editor
Two projects reproduced it on different permissions —
storage.objects.get and iam.serviceAccounts.get. A missing role fails
identically every time. Eventual consistency does not.
Conclusion: 3-5 minute IAM propagation window on new projects. The error prescribes the wrong fix.
The error tells you to fix IAM. The fix is to wait.
Persona A
Takes it at face value. Cannot fix IAM, or does not know how.
Concludes the codelab is broken, or that they lack access.
Plausible abandonment point
Persona B
Knows how to grant roles. Grants three. Next attempt succeeds.
Draws a false causal conclusion and writes it into a runbook.
Silent, and it scales
This study's analyst made exactly the second error, and caught it only because a
pre-existing roles/editor binding happened to be visible in the output.
R1: auto-retry, or detect project age and say 'permissions may still be propagating'.
Clean up reports success and leaves 356 MB billing.
Step 9 as published. 1 image deletes work. 2 a note concedes
leftover source. 3 this one-liner fails in Cloud Shell. 4 service delete works.
Audit after all four cleanup commands
Cloud Run service
deleted
Container image tags
deleted
Artifact Registry repo
356.162 MB
Orphaned source tarball
1.5 MB
Enabled APIs
29
Project
never mentioned
Repository updateTime is after both deletions —
this is not stale metadata. Deleting the tags did not reclaim the layers.
The step's implicit contract is 'you will not be charged'. It does not deliver that.
The cleanup command fails in the environment the codelab tells you to use.
Persona B, wide local terminal: worked, deleted two objects.
Persona A, Cloud Shell: failed 11 times, deleted nothing.
1 {print $4}' | while read line; do gsutil rm $line; done", "k": "cmd"}, {"t": 0.8, "s": "CommandException: The rm command (without -I) expects at least one URL.", "k": "err"}, {"t": 1.1, "s": "CommandException: The rm command (without -I) expects at least one URL.", "k": "err"}, {"t": 1.4, "s": "CommandException: The rm command (without -I) expects at least one URL.", "k": "err"}, {"t": 1.7, "s": " ... repeated 11 times, once per build. Zero objects deleted.", "k": "note"}]'>
transcript replay — not a screen recording
0:00run exactly as printed
0:01gsutil receives an empty string
0:02one failure per build, 11 total
—codelab narrative implies this cannot fail
Cause, found by Persona A using the codelab's own commented-out diagnostic line:
in Cloud Shell's narrow pane gcloud builds list renders one line per field,
not per build. awk '{print $4}' reads column 4 of a table that isn't there.
The command is terminal-width dependent.
R3: replace with --format='value(...)'. Text-table scraping is fragile by construction.
What the newcomer believed they were leaving behind.
“some small ongoing Cloud Storage charge (a handful of MB of tarballs from two builds)
is the only likely leftover”
Persona A log, written before any audit
actual
356 MB Artifact Registry
1.5 MB unreachable orphan
29 APIs still enabled
1 project, never mentioned
Wrong by roughly two orders of magnitude, in the direction that costs money.
A confident, incorrect mental model scores zero on both measurement axes —
and is arguably the worst outcome in the study.
This is the limit of the metric, stated openly rather than hidden.
One missing variable, one deeply misleading error.
${GOOGLE_CLOUD_PROJECT} is injected by Cloud Shell. On a local machine
it is empty. The shell does not warn.
// as printed in the codelab
gcloud run deploy --image=gcr.io/${GOOGLE_CLOUD_PROJECT}/monolith:1.0.0 --platform managed
// what the shell actually sends
gcloud run deploy --image=gcr.io//monolith:1.0.0 --platform managed
// what Google Cloud says back
Permission "artifactregistry.repositories.downloadArtifacts" denied on resource
"projects/monolith/locations/us/repositories/gcr.io" (or it may not exist)
The collapsed path parses monolith as the project name. The error blames
your account for a project you never created, never mentions the variable, and hedges with
“or it may not exist.” There is no path from this message to the cause.
R6: use $(gcloud config get-value project) — correct in both environments and self-documenting.
Documentation drift, visible on the page.
Step 5 as published, captured 26 July 2026 from a page dated 20 March 2026.
Page shows
Reality
1${GOOGLE_CLOUD_PROJECT}
Cloud Shell only
2 5 regions listed
44, alphabetical, no default
3 “Pod status”
Cloud Run has no Pods
4 output dated 2019
—
5**Note:**
markdown renders literally
The page says “select the region closest to you.” The prompt offers 44
alphabetical entries with no default, no latency hint and no geolocation. Concept-count scores this
zero load — “region” was already introduced — which is precisely the metric's blind spot.
R8: mark a default region or order by latency. A 44-item alphabetical list is not a choice architecture.
What the codelab gets right.
Zero-downtime deploy genuinely works. 9 seconds, HTTP 200 throughout,
three revisions retained. A real platform strength, taught honestly.
gcloud run deploy self-heals. The Cloud Run API is never enabled by
any instruction — the deploy command detects and enables it, rescuing both personas silently.
The time estimate is honest. Advertised 39 min; Persona A took 30m adjusted,
40m raw. Most tutorials' estimates are fiction.
A 2019 React app built clean on Node 26. No legacy-OpenSSL failure,
which was a plausible risk.
Concurrency and revisions are taught well — fastest step for both personas,
a differentiated capability in one flag.
Step 3 Web Preview, captured live. The codelab delivers a working app — this part is not in doubt.
A study that only finds faults is prosecuting, not measuring.
Nine recommendations. Five are trivial text edits.
#
Recommendation
Owner
Effort
R1
Auto-retry or explain transient permission failures on new projects
Cloud Build / gcloud
Medium
R2
Bring Cloud Shell error verbosity to parity with local gcloud
Cloud Shell
Medium
R3
Replace awk-scraping cleanup with --format="value(…)"
Codelab
Trivial
R4
Add Artifact Registry deletion; state the true end state
Codelab
Trivial
R5
Recommend project deletion as terminal cleanup
Codelab
Trivial
R6
Use $(gcloud config get-value project)
Codelab
Trivial
R7
Fix or remove setup.sh's local branch
Sample repo
Low
R8
Mark a default region, or order by latency
gcloud
Medium
R9
Re-capture 2019-era examples; remove Pod status
Codelab
Low
The two requiring real product work — R1 and R2 — are also the two that
decide whether a newcomer finishes or abandons.
Five trivial edits would remove most of Persona B's friction entirely.
What this study does not establish.
Persona A is a smaller model, not a human. Better than one operator narrating both
sides. Not a substitute for a study with human participants. Its timings are measurements of a model
following a protocol.
Sample of one run per persona. Two projects, one account, one afternoon,
no Organization. Enterprise IAM would change several findings.
The propagation window was not bisected. “3–5 minutes” is bounded
by observation, not measured.
One pre-registered hypothesis was wrong. This study predicted gcr.io
and --platform managed would be stale. Both still work. Recorded as a miss.
No screen recordings exist. Recording required photographing a browser window on a
macOS Space the tooling could not reach without Accessibility permission. The replays in this deck are
built from real captured output and real measured timings, and are labelled as replays —
not presented as recordings, and not re-enacted after the fact.
An entire screenshot set was discarded. 10 of 11 Persona A captures photographed the
wrong tab and were quarantined, not published. Documented rather than quietly dropped.
Stating the capture failure matters more than hiding it — it is why the visual evidence is thinner than planned.
References
Everything tested, and the documentation a reader would need to check the findings.
All links open in a new tab.
Google Cloud documentation now redirects from cloud.google.com to
docs.cloud.google.com. Links above use the destination host. Verified resolving 26 July 2026.
Every link was checked for a 200 before publication.
The codelab does not need to be rewritten. It needs to be honest about failure.
Every step works when nothing goes wrong. Both personas reached a live, public,
zero-downtime deployment. The gap is entirely in what the product says when it breaks — and in
what the reader is left believing they still owe.
5
trivial text edits remove most of the friction
2
product changes decide whether a newcomer finishes
