For about a year now I’ve had the same habit, and I doubt I’m the only one.
I ask ChatGPT a question. Then I ask Claude the same question. If it actually matters, I’ll ask Gemini too. Then I sit there with three tabs open, comparing three confident answers that don’t quite line up, trying to decide which one to trust.
That’s the part that bothers me. AI doesn’t give you the answer. It gives you an answer. Ask twice and you might get two. Ask three models and you’ll get three, all delivered with exactly the same confidence.
So I stopped copy-pasting between tabs and built the thing I actually wanted. I call it AIConsensus.
The idea is simple. You send one message. It goes to ChatGPT, Gemini, Grok, Claude, and Perplexity. One of them writes a first draft of the answer. Then the others take turns reading that draft, along with everything the rest of them have said, and each one does one of two things: it agrees, or it rewrites the answer to make it better. The moment somebody rewrites it, the clock resets and everyone has to look at the new version. It keeps going until every model agrees on the same answer.
Then you get one answer. Not five. One that all of them signed off on.
It renders like a group chat, which was the whole point. Every model gets its own logo, its own color, and its own tinted bubble, so you can see at a glance who said what. Each turn shows whether that model agreed or revised, and a one-line reason why. The final answer lands in a card at the bottom marked “consensus reached.” If they can’t agree in time, a model you pick acts as arbiter and settles it, and the card says so honestly instead of pretending everyone got along.
I built it with Claude Code over a weekend. I described what I wanted, it wrote most of the code, and I spent my time doing the part it couldn’t do for me. Which brings me to the actual reason I’m writing this.
The tool mostly worked on the first try. The bugs are what taught me something.
The first one my own eyes caught. I was reading a log and noticed ChatGPT had taken two turns in a round while Grok got skipped. The cause was boring — when a model errored out I removed it from the list, and the math that picked whose turn it was got confused by the list getting shorter mid-round. Easy fix. The interesting part is that nothing crashed. There was no error. The panel just quietly stopped being fair, and the only way to know was to read the log and count.
The second bug is the one worth writing about.
My deliberations almost never reached consensus. They’d run the full ten rounds, burn through every model twice over, and then hand it to the arbiter. Five models, and somehow they could never agree on anything. I started to wonder if that was just true. Maybe AI models are stubborn.
They weren’t. I was throwing their votes away.
Here’s how it worked. Each model had to reply in JSON: a vote, a reason, and the full answer text. Clean, structured, easy to parse. Except “the full answer text” is the entire answer — paragraphs, bullet points, quotation marks, dollar signs, line breaks. And when a model writes a thousand words with a quote in the middle and doesn’t escape it perfectly, the JSON breaks. And when the JSON broke, my parser shrugged and defaulted to “must have been a revision.”
So models were voting AGREE, and I was recording REVISE. Consensus was mathematically impossible, and I had built it that way myself.
The tell was sitting in the log the whole time. The same reason over and over: “Proposed a direct answer.” That’s not something a model said. That’s my own fallback text. My code wrote it. I had just never looked closely enough to notice my own error message staring back at me.
The fix was to stop being clever. Instead of JSON, I asked for this:
VOTE: AGREE
REASON: one sentence
ANSWER:
everything else
No escaping. No nesting. Nothing to break. The answer can contain quotes, newlines, code blocks, tables, whatever it wants, because it’s just the rest of the message. First real run after the change, all five models agreed in a single round.
If you take one technical thing from this post, take that. Don’t make a language model escape a thousand words inside a JSON string. It will get it wrong eventually, and it will get it wrong quietly.
There was a smaller one I enjoyed too. Two of my five models simply stopped existing. Gemini 1.5 Pro and Grok 2 were both perfectly valid when I wrote them down and both returned 404 a few weeks later. I’d been treating model names like constants. They’re not constants. They’re inventory. Now I ask each provider’s API what it actually has available and pick from the live list.
Then there’s the money, and this is the part I’d want any business owner to hear before they get excited about multi-model anything.
One question to five models over three rounds is fifteen API calls. And they get more expensive as they go, because every turn carries the entire conversation plus every model’s previous response. My first real deliberation cost about four cents. Four cents is nothing. Four cents forty times a day across a team is a line item.
So I built the boring parts before the fun parts. A hard cap on rounds. A cost meter in the corner that reads actual token counts back from each provider instead of guessing. Approximate pricing under every model with a link straight to that provider’s billing dashboard, because my numbers are estimates and theirs are the truth. And a stop button that actually stops — the server notices you hung up and quits after the current model finishes, instead of politely burning through the remaining twelve calls for an answer nobody will read.
One thing I often tell business owners when I consult on AI is that the impressive part is never the hard part. Getting five models to talk to each other took an afternoon. Making sure it couldn’t quietly cost you sixty dollars while you were at lunch took longer, and matters more.
There’s one design decision I’m oddly proud of. The turn order is shuffled on every single message. Whoever goes first writes the first draft, and the first draft anchors everything that follows it — the other four are reacting to it, not starting fresh. If ChatGPT always went first, this wouldn’t be a consensus tool. It would be a ChatGPT tool with four fact-checkers. Now it’s a coin flip every time.
What surprised me most wasn’t that they agreed. It’s what happens before they agree.
The disagreements are the useful part. I watched Claude go through a spreadsheet another model had summarized and verify the math line by line — the room cost, the drinks, the snacks, the profit — before it would sign off. That’s not something I would have gotten from any of them alone. I’d have gotten a confident summary and no reason to doubt it. The final answer is the receipt. The argument is the product.
Which is also, I think, the honest limitation. Five models agreeing does not make something true. They can be confidently wrong together, especially when four of them are reacting to one draft that started off wrong. What consensus actually buys you is a much better chance that somebody in the room catches the obvious mistake. That’s worth a lot. It just isn’t the same as certainty, and I’d rather say that out loud than sell it as an oracle.
The biggest takeaway for me isn’t that five AIs are smarter than one. It’s that I spent an entire weekend building an elaborate machine for not trusting the first answer I got, and then almost shipped it broken because I trusted the first answer I got.
I trusted my parser. I trusted my round-robin math. I trusted that the models were being stubborn, rather than checking whether I was the one dropping their votes on the floor. The tool exists because one answer isn’t enough, and the tool nearly failed for exactly that reason.
AI didn’t build this for me. It wrote most of the code, and it wrote it fast, but it never once told me the votes weren’t being counted. That part came from reading logs at eleven at night and noticing that a sentence looked familiar because I was the one who wrote it. The models were excellent at doing. They were not the ones doing the noticing.
Same lesson as the last thing I built. Meet it halfway. Bring your own judgment, do your own thinking, and let AI accelerate the parts that don’t need you.
AI can give you five answers. It can even get five of them to agree, show you the reasoning, and hand you a clean summary with a green checkmark and a cost estimate. But agreement isn’t the same as being right. Somebody still has to read it and decide whether the room full of confident machines actually got it.
You’re still the tiebreaker.
AIConsensus runs entirely on my own machine right now — my keys, my chats, nothing leaving the laptop. It’s still early and I’m still finding things wrong with it, which at this point I’ve decided is the feature. If it sounds like something you’d use, I’d love to hear about it.