I Taught a One-Decision Model to Play Chrome Dino, and It Kept Telling Me I Was Wrong

Last week I wrote about Jev, the AI that refuses to talk to you, and about Laya, the Apache-2.0 model that showed up a few days later doing the same three tricks with published weights. I ended that piece with a fairly smug instruction: go run it on your own data and see. Two sets of self-reported benchmarks arguing with each other is not evidence, and the fix is cheap now that one of the two ships weights you can download.

So I took my own advice, and picked a problem where nobody can argue about the answer.

The Chrome dinosaur game. The one that shows up when your internet dies. A model that only outputs numbers, wired into a game that only accepts one input, running at sixty frames a second in a real browser.

That is Laya playing. Not a recording of the game with a score overlaid afterwards, and not my simulation dressed up to look like Chrome. That is the actual Chromium t-rex runner in an actual browser, being driven by a 322M-parameter encoder answering one yes-or-no question about thirty times a second.

It took me four failures to get there, and every single one of them taught me more than the working version did.

The Setup

Laya does not generate text. You hand it a state and a typed question, and it hands back a number. So the entire control loop is this:

obstacle=cactus_small dist=118 speed=13.0 w=51 h=35 alt=0 airborne=0 vy=0.0

  "Should the dino jump right now to clear the next obstacle?"  ->  0.997

If the number is high enough, press space. That is the whole robot. No reasoning, no chain of thought, no parsing prose into a boolean. One question, one probability, thirty times a second.

The reason I picked dino over something more serious is the thing I complained about in the last post. TypeSafe’s benchmark measures how often Jev agrees with two rival chatbots, which is not accuracy, it is consensus. There is no human ground truth anywhere in it. Laya’s benchmark has the same shape, just pointed the other way.

Dino has no such problem. The ground truth is: did you survive. I can fork the game state, play out both branches, and find out for certain whether jumping at this exact instant clears the cactus. Every training label is a fact rather than an opinion, and I can generate a million of them in two minutes for free. That is a better epistemic position than either company’s launch post, and it cost me nothing but a weekend.

Failure One: It Cannot Play at All

First thing I did was ask the base model to play, cold.

It scored below chance. Not “worse than expected,” but 0.468 on a balanced two-way question where flipping a coin gets you 0.5.

To be fair to Convai, their repo says this. The base checkpoints score near chance zero-shot on typed decisions, and the headline 0.766 belongs to a checkpoint fine-tuned on that benchmark’s own training split. I even quoted this in the last post. There is a difference between reading a caveat and watching it happen, though, and the gap between “a general decision engine you can point at problems” and “a base you must train on your own labelled data” is the entire purchase decision.

So I trained it. And here is the genuinely good news, the part that held up under everything that followed: I froze the 322M-parameter encoder and trained only the decision head, which is 15M parameters, about 4.7 percent of the model. It took seventy minutes on an M1 Pro laptop. No cloud, no rented GPU, no four-hour Kaggle notebook. If you have labelled data, specialising this thing is genuinely cheap, and that is a real answer to a real problem.

Accuracy went from 0.468 to 0.964.

Failure Two: 96 Percent Accurate, Dead at the First Cactus

I plugged the trained model in, ran eight games, and it died at the first cactus in all eight.

This is the failure I would put in front of anyone who evaluates models on a held-out set and calls it a day. The model was 96.4 percent accurate. It was also completely useless, because the four percent it got wrong were not randomly scattered. They were clustered exactly where it mattered.

Here is what it actually did, frame by frame, approaching a cactus:

dist=192   oracle says no    model says 0.04
dist=180   oracle says no    model says 0.32
dist=167   oracle says no    model says 0.45
dist=155   oracle says no    model says 0.52
dist=143   oracle says no    model says 0.92   <- jumps here, dies
dist=139   oracle says JUMP

It jumped thirty frames early, sailed through the air, came down on top of the cactus. Its mistakes all lived at the boundary, which is the one place a mistake is fatal.

And notice the asymmetry, because this is the important bit. If the model jumps too early, the run ends. If the model hesitates and does not jump, nothing bad happens, because the window where a jump still clears stays open for another ten to twenty frames and I am asking it again every two frames. A false positive kills you. A false negative costs you nothing.

When the costs are that lopsided, 0.5 is an absurd place to put your threshold. So I moved it:

threshold 0.97   ->  1 of 8 runs survived
threshold 0.99   ->  3 of 8
threshold 0.995  ->  4 of 8
threshold 0.998  ->  0 of 8   (too strict, now it misses windows entirely)

Same model, same weights, same everything. Only the number I compare against changed, and it went from dying instantly to playing.

Which brings me to the thing I said last week was the more interesting technical claim, and which I now think is the actual product.

The calibration is what makes that dial exist.

Laya ships with uncalibrated confidences. It says so itself, in a runtime warning printed when you load the model, which is a refreshingly honest place to put it. I fitted a single scalar temperature on a held-out split and expected calibration error dropped from 0.022 to 0.013. That one number is what lets 0.995 mean something. On an uncalibrated model, asking for “99.5 percent sure” is asking for a number that does not correspond to anything, and the whole strategy collapses.

So the marketing hierarchy is upside down. The billboard figure is the speed. The thing that actually made this work was the calibration, and it was not on by default.

It also puts the “zero hallucinations” claim in an unflattering light. Strictly true here: Laya cannot return an option I did not define, so it can never answer “banana” to a yes-or-no question. But it told me 0.92 for “jump now” at a moment when jumping meant certain death. Constrained output guarantees the answer is well formed. It guarantees nothing whatsoever about the answer being right.

Failure Three: The Bug Was Mine

Now the good part, and the reason I think this experiment was worth doing at all.

I had been training against my own re-implementation of the dino game, written from Chromium’s source constants so I could fork the state and generate labels. The model learned it, played it well, and scored around 1,987 in simulation.

Then I wired it into the real browser.

It scored 248.

My first instinct was that the model had not transferred, which is a comfortable thing to believe because it puts the fault in the model. So I did the obvious check: I took the exact frame where it died in Chrome, reconstructed that state in my simulator, and asked my oracle whether jumping there was survivable.

My simulator said yes. Chrome said no. One of them was lying, and it was not Chrome.

The bug was four lines deep in Chromium’s source. When the dinosaur passes a certain height, updateJump calls endJump, which clamps the upward velocity and cuts the ascent short. The jump is capped. I had modelled a clean parabola.

Twelve pixels. Three frames. That is the entire difference, and it was enough that my simulator believed a wide cactus cluster was clearable from 201 pixels away when in reality you had to be at 190 or closer. The model had learned my mistake perfectly and executed it faithfully at sixty frames a second.

I fixed the physics, regenerated the data, retrained. The score went from 248 to 3,209.

I want to sit on this one for a second, because it is the most transferable lesson here and it has nothing to do with Laya specifically. No metric on my dashboard could have caught this. Accuracy was fine. Calibration was fine. Loss curves were fine. The model was a flawless student of a curriculum that was wrong, and every number I had was computed against that same wrong curriculum, so everything agreed with everything else right up until reality disagreed with all of it.

If you are evaluating a model against a simulator, a synthetic dataset, or a set of golden answers you generated yourself, you are measuring agreement with your own assumptions. That is exactly the criticism I levelled at TypeSafe’s benchmark last week, and I walked straight into it.

Failure Four: The Demo Video Is Lying to You, Including Mine

The video at the top of this post shows a good run.

Here is a different run. Same model, same weights, same threshold, same settings, a few minutes apart:

I ran it 28 times and wrote down every score:

81, 240, 241, 251, 300, 368, 405, 428, 431, 464, 736, 887, 893, 896,
1057, 1068, 1079, 1112, 1128, 1312, 1765, 2086, 2178, 2190, 2552,
3208, 3208, 3209

median 976 | mean 1206 | range 81 to 3209
reached maximum game speed:  17 of 28
survived the full time limit:  3 of 28

An eighty-one and a three-thousand-two-hundred-and-nine came out of the same model on the same afternoon. That is a fortyfold spread driven entirely by which level the game randomly generated.

I nearly published the wrong number. After my first five runs I had a median of 2,552 and was quite pleased with it. After twenty runs it was 1,062. After twenty-eight it settled around 976. Each time I added data the story got less flattering, which is roughly how you can tell the early numbers were not measurements but sampling luck.

So: every “watch this AI play a game” video you have ever seen, including the one I put at the top of this post to get you to keep reading, is one sample from a distribution you are not being shown. Mine happens to be a good sample. I know that because I counted. Most demos do not tell you, and a meaningful number of them have not counted either.

For scale, most humans score somewhere between 500 and 1,500 on this game. Median 976 puts the model squarely in ordinary human territory, better than most people at its best, and nowhere near reliable.

So What Did I Actually Learn About One-Decision Models

Three things held up, and three did not.

Holding up: the latency claim is real. About 23 to 30 milliseconds per decision on a laptop, roughly 2,800 decisions in a long run, against a game that runs on its own clock and does not wait for anything. That is the core pitch and it survived contact with a hard real-time loop. Worth noting that a frame at sixty fps is 16.7ms, so on paper 30ms should be fatal. It is not, because the dinosaur does not need to decide every frame. It decides once per obstacle, inside a window I measured at 200 to 367 milliseconds. Knowing that number saved me an entire optimisation project.

Holding up: specialising is genuinely cheap. Seventy minutes on a laptop, 4.7 percent of the parameters, free labels. If your problem produces its own ground truth, this is an afternoon.

Holding up: calibration is the real contribution. I said this last week based on reading. I now believe it based on the fact that nothing worked until I fitted the temperature, and that the single most important tuning knob in the entire project was a threshold that is meaningless without it.

Not holding up: zero-shot usability. Below chance. This is a base model that requires labelled data, not a decision engine you point at a problem. Both companies’ framing obscures this, and Laya at least documents it.

Not holding up: “zero hallucinations” as a safety property. It means the output parses. It does not mean the output is correct, and a confidently wrong 0.92 kills you exactly as dead as malformed JSON would have.

Not holding up: numbers outside the training range. At one point the model confidently said 0.997 for “jump now” when the cactus was 326 pixels away, which is nowhere near jumping range, because I had accidentally starved that distance band out of the training set. Encoders are weak at numeric thresholds and they do not degrade gracefully at the edges. They go strange, and they go strange with high confidence. If you are pointing one of these at sensor readings or anything continuous, that is the failure mode to go looking for.

The Objection I Was Sure Would Beat Me

I said from the start that this whole thing was silly, because dino is solved by a three-line if-statement:

if obstacle_is_jumpable and distance < k * speed:
    jump()

I believed that strongly enough to build the baseline first, expecting to have to write a post about a 322M-parameter model losing to four lines of Python.

On a simplified dino it does win, comfortably, for any k in a wide range. On the faithful Chromium physics it falls apart, because whether a fixed jump arc clears a 75-pixel cactus cluster depends on the arc, the width and the speed together, and that is not a linear function of distance. I gave the baseline every advantage: a second parameter for obstacle width, and a full two-dimensional sweep to find its best configuration.

Then I ran both on three separate sets of levels. The model had never seen any of them. The baseline had been tuned on the first set.

The one-liner wins only on the levels it was fitted to. Tuning k until it works on the levels in front of you is fitting the test set, and it does not survive first contact with levels you have not seen.

I was wrong, and I was wrong in a way I would not have discovered without building the honest version of both.

Should You Care

If you are deciding whether to put a one-decision model into something real, the useful summary is this.

The speed is real, the cheapness of specialisation is real, and the calibration is the part worth paying for. The zero-shot story is not there, the safety story is weaker than the chart implies, and the thing most likely to break your deployment is not the model at all. It is the gap between the world you trained it on and the world it runs in, which no accuracy number will ever show you.

My model did not fail at dino. It played my imaginary dino perfectly. The imaginary dino was the problem.

Code, data generation, training and the browser bridge are all in laya-dino on GitHub, including the run log with every dead end in the order I hit them.