Blog post #54

---

My friend Martin is prepping for an Azul tournament. He’d been practicing against the bots in my little Azul-coach web app — and beating them too easily. So I set out to give him a real opponent: a bot that actually learns the game by playing itself, AlphaZero-style. What followed was three weeks of one machine-learning lesson after another, most of them learned the hard way. The bot is now live, it’s called Cobalt, and it’s measurably stronger than where I started — though Martin can probably still take it.

---

What shipped

• A complete AlphaZero pipeline for 4-player Azul, written from scratch in Python: a state encoder, MCTS, self-play game generation, GPU training, and numpy-only serving so the web app needs no heavy dependencies.
• Cobalt — the deployed bot — went through generations: heuristic → az14 → az20 → finally az59. Each step measurably stronger than the last.
• Every finished online game is now logged to Supabase, which doubles as training data.
• It’s live: you play 1–3 copies of Cobalt, and the move log, the “Cobalt won by 4” banners, everything is named after the cobalt pigment that makes azulejo tiles blue.

What’s working

• Self-play actually works. The bot has now played itself something like 17,000+ four-player games — more Azul than any human plays in a lifetime — and it climbed from “dumps tiles on the floor” to beating my hand-written heuristic and a 1000-iteration MCTS.
• The final breakthrough — Gumbel AlphaZero. After the plateau (below), a deep-research pass pointed me at a 2022 paper: standard MCTS (PUCT) can provably fail to improve its policy when the simulation budget doesn’t visit all the root moves — which is exactly my situation (100+ legal moves, a few hundred sims). Gumbel AlphaZero guarantees improvement even at very few simulations. I implemented it; self-play got 5–10× faster, which let me run a much bigger data campaign. The result, az59, beats the previous champion az20 35% of the time in 4-player head-to-head (a fair share is 25%) with a higher average score. First real progress in weeks.

What’s unclear or broken

• The first collapse. Pure self-play from a random network collapsed into a degenerate strategy: every bot learned to dump its tiles on the floor, everyone tied at ~0 points, and the value signal flatlined. The fix was the AlphaGo move — imitate a hand-written heuristic first to get a competent starting point, then let self-play take over — plus a rank/placement-based value target that can’t collapse to a constant.
• The plateau. Then came four straight training rounds that went nowhere. A richer state encoder: no gain. Dropping a bias in the training data: no gain. A 4× bigger network: actually worse — it was undertrained for its size, not (as I first assumed) some exotic failure mode. Four rounds, ~az20 every time. That was humbling, and it’s what sent me to the literature.
• Martin is still probably better. Honest about the ceiling: az59 is a modest improvement, and a real leap to expert-human level needs orders of magnitude more compute (~500 Elo per 10× of compute, per the research). We didn’t leapfrog Martin. We just stopped losing ground.

Decisions made

• Trust the research over my instincts. My instinct was “bigger network.” The research said the opposite: with a fixed data budget, a bigger net just starves. The lever was more data, and Gumbel was what made more data affordable. I’d have wasted days guessing.
• A modest, measured win is a real win. It would’ve been easy to deploy something on a noisy 20-game result that looked great (one candidate showed 54%, then fell to 27% over 40 games — pure noise). I made it a rule: confirm on 40 games before shipping. az59 held up; the flukes didn’t.
• Don’t deploy a regression. Several rounds I chose not to ship, because the new net was only equal to the old one. The bar was “beats the current champion head-to-head,” not “is new.”

Tooling & process

• The whole thing is human + machine, end to end: I worked through the design, the bugs, and the strategy with an AI agent, and the bot itself learns purely from playing against copies of itself.
• Training runs on a single consumer GPU; the bottleneck is CPU-bound self-play, parallelized across cores. A “deep research” agent fanned out across ~25 sources and fact-checked the claims before I built anything on them — that’s what surfaced Gumbel.
• The unglamorous battles took real time: Windows kept sleeping mid-training because its idle timer ignores CPU load (fixed by asserting “system required”); transient GPU crashes (fixed with step retries); and background jobs dying whenever I closed the app (fixed by making the whole loop resumable — it skips finished work and continues).

The bot got better in clear, measurable steps, and I finally understand which knob actually moves the needle. Next time Martin wants a harder opponent, I know exactly what to do: more self-play, not more cleverness.

---

— Stefan