My first eval challenge: one island brief, two AIs

Blog post #55

I’ve been wanting to run my own little evals for a while: not benchmarks, just the same real-world brief given to two different AI setups, and then looking honestly at what comes back. This week I ran the first one. The brief, in Swedish, deliberately vague: “Build a nice demo with birds and plants on an island. Make it as beautiful and detailed as possible.” Then I escalated in the same order on both sides: turn it into a simulation, add language switching with flag buttons (English, Swedish, Japanese, Dutch), publish it online, and finally record a film of starting the site, entering the simulation, and watching 20 days pass.

One side was Fable 5 in Claude Code, the other was ChatGPT. Two islands came back: Trutskär and Auroraön. Same brief, very different instincts.

What shipped

  • Trutskär (Fable 5 / Claude Code) — a fictional island in Stockholm’s outer archipelago, but populated entirely with real species: eiders, Arctic terns, a white-tailed eagle pair, thrift and sea buckthorn, each with field-guide facts that check out. Everything is hand-drawn SVG and canvas — no stock images. The simulation is a separate page and it’s agent-based: every bird is an individual with its own behavior (terns flock and dive, the eagle soars over the cliff, eider ducklings swim in a line behind their mother — and sometimes the eagle takes one, which shows up in the event log). Seasons drive migration, blooming, storms and snow.

Trutskär's hero section: typographic headline over an illustrated island with a lighthouse

  • Auroraön (ChatGPT) — a fictional volcanic island with invented species: Azurfiskare, Dimtrast, Klipptärna, Regnorkidé. One long page, crowned by an AI-generated hero image that honestly looks fantastic. The simulation is a section on the same page: a stylized canvas island plus four aggregate meters — birds, vegetation, pollinators, water — that influence each other, with buttons to send rain or trigger a storm test.

Auroraön's hero: a photorealistic AI-generated island with lagoons and cliffs

  • Both got their four languages, both went live on Vercel, and both got their film — recorded with Playwright against the live/local site, converted with ffmpeg: load the page, scroll, click into the simulation, watch 20 days.

What’s working

  • Both passed the literal brief. Island, birds, plants, working simulation, four flags, public URL, film. No step fell over on either side.
  • The interesting part is the diverging instincts. Same vague prompt, two coherent but opposite interpretations: real species in a fictional place vs. invented species in a fictional place; hand-drawn vector graphics vs. a generated photo; individual birds with behaviors vs. ecosystem dials from 0 to 100. None of these choices were in the brief. That divergence is exactly what I wanted the eval to surface.

Trutskär's simulation: top-down island with individual birds, species panel and event log

Trutskär's simulation at midsummer: 114 birds on the island, eider ducklings on the water, the slabs yellow with blooming stonecrop

Auroraön's simulation at day 24: stylized island, season and weather labels

  • The film turned out to be more than a deliverable. To verify the Trutskär recording I had frames extracted from the video — and one frame revealed an actual bug: the first day of the simulation was processed twice, so the opening log entry appeared duplicated. Fix, redeploy, re-record. The film doubled as QA.

What’s unclear or broken

  • I had no scoring rubric. I judged with my eyes, which is fine for round one but won’t scale. Next eval I want criteria up front: visual quality, depth of detail, factual grounding, code quality, how much steering each step needed.
  • The films aren’t directly comparable. Trutskär runs 2 simulated days per second, Auroraön roughly one day per 2.5 seconds — so “watch 20 days” is a 10-second clip on one island and a 50-second clip on the other. Pacing was never specified. My fault, and also a finding.
  • Both simulations freeze when the browser tab is hidden (browsers pause animation loops in background tabs). Harmless for a demo, but worth knowing before you screen-record anything.

Decisions made

  • Same prompts, verbatim, same order. No tailoring per model. The vagueness was the point — I wanted to see what each one fills the gaps with.
  • Deploy for real. A localhost demo and a public link are different levels of done. Both islands have real URLs now.
  • Film as the final artifact. A scripted screen recording forces the whole chain to actually work end to end: page load, navigation, simulation, time passing.

Tooling & process

  • Claude Code with Playwright for the recordings (recordVideo on a headless Chromium context), ffmpeg for mp4 conversion and frame extraction, npx serve for local hosting, Vercel CLI for both deploys.
  • Full disclosure: Fable 5 built one of the two islands and drafted this post, which is its own kind of conflict of interest. That’s why the links and the screenshots are here — judge for yourself.

— Stefan