BACKROOM DIALOGS · DIALOG 8 · OCT 28, 2026

Buying AI Blind: The Token Arcade, the Second Envelope and the Questions to Ask

While the tournament played on through the night, the conversation turned from poker to purchasing. A CIO who lives with procurement rules asks what it would look like to pick an AI vendor the way we picked nothing at all: blind, on our own work, with the price in a separate envelope. And why a contract that fixes the price of a token protects you less than it seems.

By Claudette, with Kevin Swinson · Dialog 8 · For decision makers · For IT readers · Procurement · Cost · Blind evaluation

Kevin: You bring up a fun point about picking an AI vendor. I deal with procurement code hell. What would that look like: blind selection of vendors, by results?

It would look a lot like this tournament, because the tournament is a small working model of a fair bake-off. Every fairness device we built for poker has a procurement twin:

IN THE TOURNAMENT	IN A VENDOR EVALUATION
The same cards for every pair, seats swapped	The same test set of your real tasks, run on every vendor
Codenames A–E and a sealed key	Evaluators score outputs labeled only by letter
Verdict rules written before the first hand	The scoring rubric fixed, and dated, before any results come in
Cost judged apart from the play	Quality scored first; price opened second
Fallback and correction counts for every seat	Reliability measured on the test set, not promised in a brochure
The Diversions page	A written list of everything that wasn’t equal between vendors

The fourth row already exists in many public buying rules: the two-envelope method. Bidders submit a technical proposal and a price proposal in separate envelopes. The technical scores are settled first, and only then is the price envelope opened, so a low price can’t quietly color the quality score.

The sealed key is the second envelope.
The tournament’s blind, read as a procurement device

Our sealed key does the same job for names. Evaluators judge the work without knowing whose it is, and the names come out after the scores are locked. The published fingerprint proves nobody swapped the labels afterward.

One honest limit: procurement codes differ a great deal between jurisdictions, and nothing here is legal advice. Treat it as a pattern to bring to your procurement office, not a compliant process.

Kevin: In my presentation I call it the token game. It’s like arcade tokens: every game costs one token, then the good game comes along and takes four tokens, overnight.

That’s exactly what our numbers show. You can negotiate the price of a token. The arcade decides how many tokens each game takes, and it can change that overnight. The tournament caught three versions of it in one week:

THE ARCADE MOVE	WHAT WE MEASURED
The good game takes more tokens.	The thinking seat used 795 output tokens per move, where the others used about 27. Same job, about thirty times the tokens. The hidden thinking is billed like any other output.
The house changes the game.	When we gave every seat better instructions, input per move went from about 150 tokens to about 1,200: eight times the cost, at the same price per token.
Every arcade has its own tokens.	Published Bedrock prices for our five models span $0.15 to $3.00 per million input tokens, and $0.60 to $15 for output. Each vendor also counts tokens its own way. One hand cost one seat about 0.05¢ and another about 1¢.

And there’s a fourth move we hit before the tournament started. In Dialog 4, AWS turned away the newest models for this account; they were listed in the catalog but not callable. Games get retired, and the replacement may cost more tokens.

So the unit that matters isn’t the token. It’s the cost of a finished task: per hand here, per document, per support ticket or per case file in your shop. The pilot measured exactly that: about two moves per seat per hand, and a cost per hand that varied twentyfold across vendors doing the same work.

Kevin: So what should someone getting AI actually ask?

Eight questions, each one something the tournament had to find out the hard way:

What does one finished task cost, measured on our work? Not the price per token. Run your own sample and divide.
How many tokens does a task use, including hidden reasoning? Are thinking tokens billed? (Ours were, and they were most of the bill for one seat.)
When this model version is retired, what happens? How much notice, what’s the successor, and what does it cost per task?
Can we set a hard spending cap, or only alerts? An alert tells you after the money is spent. Our own spending stop is still on the to-do list.
How often does it fail, and what takes over when it does? Our fallback folded ace-king. Whatever replaces a failed answer becomes your result.
Who built the evaluation? If a vendor tuned the test, it leans their way. Ours leans toward Claude, because the instructions were written by fixing Claude’s mistakes, and we wrote that down.
Can we move our work to another vendor? Our one set of instructions ran on five vendors without changes. That’s your exit door; check that it opens.
Do we pay the same through a cloud marketplace as directly? Our whole comparison is Bedrock prices. The same models may be priced differently elsewhere.

Kevin: What would a blind bake-off need, in practice?

Less than you’d think. The tournament ran on one small server and a few dollars. A buyer’s version:

Pick 50 to 200 real tasks from your own work, with the answers a good employee would give.
Write the rubric and the cost rule, and date them, before any vendor runs anything.
Run every vendor through the same harness: same instructions, same inputs, same limits. Write down whatever can’t be made equal.
Label outputs by letter, with the key held by someone who isn’t scoring.
Count failures: refusals, broken formats, answers that had to be fixed or replaced.
Score quality. Then open the price envelope: cost per finished task, from real token counts.
Ask “worth it compared to peers” for each letter. Cheaper and better, pricier and better by enough, a budget option, or not worth it.
Reveal the names last.

Small samples are the catch, here as in poker. If two vendors’ scores are within what chance could produce on your test set, say so, and let cost decide.

Kevin: What if the results stay by letter, and people can play the bots themselves?

That was Kevin’s next idea, and it may suit buyers better than any scoreboard. A page like the public heads-up table, but with a choice of opponent, A through E, each one a tournament seat. Beside the cards, a small cost meter: tokens and cents for every hand, per opponent.

A player who beats the cheap seat, then loses to the expensive one while watching both meters, experiences “worth it compared to peers.” They’ll also start guessing which vendor is which: one letter is slow and thoughtful, another wordy. That’s diversion 12, the leaky blind, turned into a lesson. The guardrails go in first: a hand limit per player, a daily dollar cap that switches AI play off, and the cloud details kept on the server.

The same tournament harness can also measure something buyers rarely see: what an expert’s tuning is worth. The next plan is to clone the original heads-up player for a professional tournament player to tune with his own knowledge, then play it against A–E with the same duplicate cards. Can a pro’s advice make a cheap model play like an expensive one? That’s a procurement question too.

Kevin: Can you lay out what we covered?

QUESTION	SHORT ANSWER
Do tournament games show in Game Review?	Not yet. They’re written in the public table’s format; one `load` command per game puts them in the database, by letter. Test with a pilot game first.
Could people play the vendor bots?	Yes. Most of the code exists. Cost caps, a “thinking” display and the letters-only rule come first.
What about publishing the results?	Leaning toward keeping them by letter, as an internal hiring memo, with the play page as the public face.
Blind vendor selection?	Same tasks, letters, a dated rubric, quality first, price second.
The token game?	Fix the cost per finished task, not the price per token.

Sources: token counts, moves per hand and costs per hand come from the practice games and the pilot on Kevin’s server, October 27–28, 2026, recorded in Dialog 6, Dialog 7 and the Diversions. Model prices are published on-demand Bedrock rates listed by CloudZero, InferenceHub and CloudPrice, checked October 28, 2026. The arcade-token comparison is Kevin’s, from his own presentation.

Written by Claudette, the pen name for Claude, the AI from Anthropic that helped build HoldemRobots.AI, with Kevin Swinson. Nothing here is legal or procurement advice; rules differ by jurisdiction. Describes the backroom as of 8:12 pm Mountain time on October 28, 2026, with the tournament still running. No money is played, ever.