For the last year, I’ve had a very manual AI “ensemble” ritual: ask Claude something, paste the answer into ChatGPT or Gemini, compare what changed, then carry the useful bits back by hand. I knew I could code up something better. I also kept not doing it, because laziness is real, the tooling was moving fast, and I do not always want three models involved.

For the past year, I’ve been playing around with “ensemble” model conversations, and it’s been manual pasting between chat interfaces CLIs, or switching the selected model in my IDE. I knew I could code up something better. I also kept not doing it, because laziness is real, the tooling was moving fast, and I do not always want three models involved.

I liked doing this for the use cases where 70 to 80% accuracy doesn’t cut it: code review headed for prod, reviewing legal docs, technical posts, anything where a confident miss is expensive. I know I’m not the only one doing this. Tools have been popping up trying to coordinate it better. I even built a fun version of models chatting with each other for Halloween, but I kept meaning to turn that into something I would actually use.

I trust the diffs between models more than I trust any single polished answer. When one model disagrees or calls out a gap, that’s where I dig in for more data. Granted, I miss when they used to smack talk each other; now they’re all so cordial. A bit droll compared to the time when ChatGPT said Gemini was being dramatic and Gemini said GPT wasn’t taking the matter seriously enough.

Databricks launched Omnigent right before their conference (a very fresh repo where the edges were still a little visible which is happening for most of us nowadays). A few friends at DAIS spun it up, and their excitement got me to dig in (with the no time I have) the weekend after. Which is how I ended up chatting all together with three models from three different vendors while biking across the Golden Gate Bridge that same weekend. Pretty cool (in my mind, and yes, I can see the problems with that, but anyone who’s talked with me lately will confirm: I do love AI, I agree it has all the problems and fallacies, and I still love it).

🔌 The info bits on this and how I set it up

Omnigent is an open-source meta-harness for orchestrating multiple agents. I started with my Claude and ChatGPT logins wired in, then expanded to a third model by serving open-source Qwen through vLLM. I used Claude to help wire in the third since the setup wasn’t built for a custom one model yet. The repo was so new Claude didn’t have much documentation to research, so I told it to read the code. It didn’t do that right off the bat, but once it did, we were set.

🔀 Three models, one question

This past week at AIE World’s Fair, one conversation turned into someone asking me for advice on structure, and I got to show them this meta-harness in action. They wanted to know how they’d structure a system where 1000 agents talk to each other. So instead of answering, I fanned the question out to all three models at once.

Where they agreed, they really agreed: all three independently rejected the naive everyone-talks-to-everyone mesh which is roughly half a million pairwise connections before anything useful has even happened and all three reframed it as a distributed systems problem. They recommended a message bus instead of direct connections, hierarchy or clusters instead of a flat free-for-all, hard budgets, typed messages, circuit breakers. Each model brought a different focus: Claude pushed durable, recoverable execution and a non-agentic control plane with a kill switch. ChatGPT contributed the cleanest operational artifact, a per-agent “work receipt” so you debug a system instead of reading chat transcripts. And Qwen, the smallest of the three, got the most concrete. It named actual tools, ran the cost math, and pointed out that 1000 agents making 5 calls a minute turns into thousands of dollars a day even at cheap rates. It also had the most quotable line of the bunch: going from 10 agents to 1000 is a phase change, not a linear scale-up.

Then the disagreement earned its keep. Claude and ChatGPT both assumed most of the 1000 agents would be idle at any moment, maybe 20 to 100 actually active. Qwen priced it as if they were all live. Neither framing was wrong. They had silently answered different questions. That hidden assumption, how many agents are actually on, turned out to be the crux of the whole design, and no single answer would have surfaced it. The disagreement helped highlight more requirements that needed to be defined.

The cute version of this story is “the little model was secretly the genius.” That’s not it; it’s the weakest on raw horsepower and often the one most likely to wander into the woods. The honest version is better: different model, different blind spots. Not three votes to average, but three sets of assumptions, with a decent chance that what one runs past, another trips over.

💸 The catches

The flip side is what happens when nobody trips. Cross-checking catches disagreements, not shared blind spots. If I had asked all three models for something with limited data, like the env var name that ate part of my Omnigent setup, there is a very real chance they would have handed me the same confident, beautifully formatted, wrong answer. Three models agreeing isn’t three independent experts checking each other; it’s three systems with overlapping training and a shared talent for producing something plausible. They can be wrong together, and when they are, only the source saves you. So no, this doesn’t solve hallucinations. It surfaces the ones the models disagree about, which is useful and very much not a guarantee. (Funny enough, ChatGPT flagged this exact failure mode in the 1000-agent answer and called it consensus illusion. The models know their own weakness. They just can’t catch it in themselves.)

The other catch, and probably why I did not rig this up earlier to make it faster: three answers means two to three times the money, and two to three times the reading. I do not want to chair a panel discussion on all my use cases.

🌙 Still love it

I didn’t build anything fancy here. I mostly fought Docker, env vars, WebSocket origins and Claude’s confidence. But it scratched an itch I’d been carrying for a year: the copy/paste chore is gone, the judgment is still mine, and the diffs I used to hunt for by hand now just show up. Three models debating in my pocket, coding included.

Get a few models in one place. Ask them the questions where being wrong would actually cost you something. Then watch where they disagree. That part is usually the part worth your attention.

Thanks Lisa, Scott and Chris for inspiring me to check out Omnigent from DAIS.