Finally Diving into AI Model Evals

I’ve been asked in the past how we evaluate our AI models at Fight Health Insurance (FHI), and frankly, it was pretty bare bones, and from what other startup founders have told me, they weren’t far off from that either. We didn’t have the time or resources to dive in, and the models in 2025 weren’t at a point where they could help us out the way we needed. At the end of August I finally had the time and resources to go down the eval path.

What started as dipping a toe in ended with me falling in head first in an evolving approach: 28 models in 35 configurations, each configuration writing 632 appeal letters, about 21,900 non-empty letters and 112,000 judgments in all. I counted each model once, but treated a different quantization or prompt as a separate configuration. All this would have cost about $4,250 and we paid about $450 in cash and used about 490 million tokens, over about a week and a half. For perspective, Anthropic’s published average for a developer on an active day with Claude Code is about $13, which is somewhere around 25 to 40 million tokens, so this was 2 to 4 weeks. And for someone running several agents in parallel all day, it’s a day or two.

What came out of it is what we run now, a 4-bit Gemma 4 26B with a two-sentence prompt change, at under four seconds a letter. What was more interesting to me was how differently the judges scored the same letters. So… I’m going to share below more about what I did, how I did it, what the results look like and what I would do differently.

Origin Story

In case you haven’t heard, FHI helps people write appeals when their health insurer denies a claim. We fine tuned (adapted through a type of exposure therapy) multiple large language models to help with this task. In 2025 after we fine tuned different models, we ran them with a handful of example denials and read the letters they produced to see if they read like a solid appeal letter. We would check them from time to time by running them ourselves mostly in prod. We had a small set of generated and curated examples that we would run through, but we didn’t have the time to build it out much further.

Evaluating an AI model can take different forms, and it’s top of mind for most companies putting AI into prod. Ordinary software tests help me check the harness. They don’t tell me whether it wrote a good appeal. The same input doesn’t give the same output every time, so you need a different way to measure success and build enough confidence to ship. I wanted a way to test open models and models we made on generating healthcare denial appeal letters. An eval suite also catches regressions and shows how a model handles our field of focus. Most of all it gives me structure. I can say I ran it against the set and here is what changed, instead of it feels better.

I got to dive into evaluating the models we made, and many others, with all the agent power we have access to right now (all the power, limited time only), including getting Claude and Codex to talk directly with each other with no other structural help. Initially my goal was to run our fine-tune models against their base models, then see how they compared with some of the newer models on the scene for our use case. As I worked through the evaluation, the list of questions grew to things like:

• Are there newer models we should start to use in FHI?

• Does a quantized model perform as well as its full size version?

• Which judges are best to use for evaluations?

• How should I structure future evaluations?

• Should we fine tune another model and any changes to that process?

• What are evaluations not good for?

It could have still been better structured from the get go but I do like to build and learn. I got some surprising results, some less surprising ones, and I’m still learning. This post is my way of sharing what I’ve learned so far.

The Data

Claude and I assembled the eval set from 600 state-review records we had already amassed for FHI, from New York, California, and Washington, plus 33 rows of our own (e.g. stress cases, cases we’d written last year). That’s 632 unique cases. The state records are the outside reviews each state runs when a patient takes a denial past the insurer’s own appeal process (CA calls it Independent Medical Review, IMR). They show the denial, the appeal, and the review board’s decision and reasoning, whether the denial was overturned or upheld. They varied in content from useful narratives to mostly medical codes. That range was good because a model that writes well with a full story still needs to handle a denial that gives it very little to work with.

The tables add up to 633 because one New York case slipped in twice for some configurations before it was removed, so the 600 state rows are 599 unique cases. Every configuration wrote letters for the 632 unique cases.

The AI model writers only got the denial and any case background it included. The AI model judges got more: the review board’s decision and reasoning, labeled so they knew the writer never saw it.

It’s important to note the proper methodology says the eval set stays out of training and fine-tuning, and in the case of our fine tuned models, they had already been trained on this data. I accepted the muddy waters when it came to measuring quality because I wanted to still explore what we could get with what we had available. There were still some insights to gain on where the models were struggling and what I needed to investigate next.

The Criteria

Quality, speed, and cost were top of mind for the evaluation criteria. Granted I was adjusting that criteria a bit as this developed so also not a perfect setup. Sounding like (masquerading as) an appeal letter was not enough.

Quality criteria:

• addressing the denial,

• making the appeal reason (e.g. medical-necessity) argument that is factual

• avoiding invented facts in details and citations,

• producing a usable letter (tone and form).

Each quality criterion scored 0 to 2, with cost and speed recorded per letter alongside. The quality scores were combined into a total from 0 to 1, with one hard rule I call the fabrication gate: if a judge scored a letter zero for invented facts, that judge’s whole score for the letter went to zero, no matter how good the rest of it was.

I scored every letter two ways. The rubric score is the judge grading the letter on its own, and that’s what’s in the tables. The head-to-head puts it next to a reference letter for the same denial and asks which is better, because rubric scores bunch up: most of the field sits between 0.8 and 0.95 on the lenient judges, and a judge that gives everyone 2s can still pick between two letters. Its numbers aren’t here. With one anchor for nearly every model it mostly says who beats Gemma 3 27B (the untuned model our fine-tunes descend from), not who beats their own base, so it’s a check I’d use rather than a ranking I’d stand behind. It does show up in the cost though: about half the judging went to it.

Additionally, I decided to track how fast it came back, how many tokens it took, and what we paid for it. I didn’t weight any of this during the run. The harness recorded quality, speed, tokens and cost separately, and most of the tables here show them that way. For the full listing at the end I picked weights so it could be sorted: quality counts for half, then speed, output length and cost. That’s my judgment about what matters for our use rather than something the eval decided, and every column is there if you’d weigh it differently.

The Setup

Claude, Codex and Qwen helped me hand-roll an evaluation harness, about 600 lines of Python and no eval framework, that worked like this:

• Ask mode to generate appeal letter

• Judge appeal letter

• Report results

• Record every run in a manifest, a JSON file that says which model ran, which judges scored it, and when

• Store every intermediate result in a JSONL file for review

We served the open-weight models with vLLM, on FHI’s own 5090 or on the H100 Azure donated when they fit. Everything else went through an API. The harness talks to anything with an OpenAI-compatible endpoint, so both looked the same to it. As I worked through my initial question and approach, it evolved, I added on more things to track and ended with each letter getting recorded with latency, tokens, errors and cost, per token for APIs calls and per GPU-second for self-hosted. Some models ran before I had their prices in the harness’s price table, so I priced those afterward from the token counts. I also had the models assessed in their quantized state as well as using tighter prompts.

We added a tight prompt experiment and quantization in that last round where we worked with the top scored models and applied this prompt to drive improved results. “Output ONLY the letter itself: no markdown formatting or headings, no commentary before or after, no notes to the user. Plain prose, and end immediately after the signature block.”

Judging was all API calls. Every generated letter was already saved as a file next to its denial, so I never regenerated letters to send to a new judge. Each batch of saved letters went out to a judge’s API, and everything resumed by id, so a crash or a new judge picked up where it left off.

The Writers (Evaluated Models)

My first report covered 8 models. I kept adding comparisons: base models against our fine-tunes of them, newer models, 4-bit builds, and the tight prompt to better scope the output. That landed the whole evaluation at 28 models in 35 configurations, since a different quantization or prompt counts as another configuration but not another model.

Some writer models got cut along the way, and two got added late. The biggest API models stopped being candidates when I realized that we weren’t hosting them and their evaluation results in relation to others didn’t make the case to keep reviewing them. Kimi-K3’s weights are 1.5 TB and GLM-5.3’s 756 GB, so they stayed in as reference points.

Two tight-prompt configurations never ran, Qwen 3.8 27B because the vLLM image couldn’t load it and GLM-5.3 because other models showed higher signal to focus on. Mistral-Small 3.2 24B was a late entry that we evaluated because the only Mistral in the single-GPU class was a 2024 7B (Mistral Small 4 is a 119B mixture that doesn’t fit a 5090 or H100). And DeepSeek V4.1 Flash came out on Sep 10, after the main run, so it went in while I was writing this up, since it’s a fraction of V4-Pro’s price and I wanted to know what that costs in quality. It was interesting to go through the process of adding and removing these models as we got more clear on what our needs were and which judges could address them.

The Judges

Of course I used AI models to judge the AI model writers. The first two real judges were DeepSeek V4-Pro and Claude Opus 4.8. I wanted different model families to get more clarity on pain points in the data. An earlier smoke test (a throwaway first pass with one writer and one judge, just to prove the harness worked) had already shown how lenient a single judge could be, and reading a few by hand showed me the rubric needed to handle letters that came back with unfinished [bracketed] placeholders.

As I worked through the process I added more judges for more perspective, and to cover the letters written by models that were also judges, since no judge scores its own letters. I added Jev, TypeSafe’s model, as we ramped up. It received the writer’s prompt and letter, with questions covering the same four quality criteria. For each one it returns a probability across the rubric levels plus a confidence, and I stored only the expected value. Those four averaged onto a 0 to 1 scale, with no fabrication gate. Keeping the distribution is what would have let me set a fabrication threshold and check it against my own read, and that’s the choice I’d revisit.

We landed with 5 AI judges, Jev and me. The Where column is who paid for it. The last column is how often each judge said a letter made something up. For the five LLM judges, a fabrication flag zeroes that judge’s rubric score on a letter, and it’s where the judges split. From here on I call the ones that rarely flagged lenient and the ones that flagged a lot strict. My blind read, under The Results, is how I got an initial signal on which of them were objecting to quality that were actually positive points we wanted from the models.

Every LLM judge scored the same rubric: cites the denial, argues medical necessity, invents no facts, tone and form. The key points to know about the judges and process:

• No judge scored its own letters. That’s why GPT-5.5’s letters use GPT-5.6, and DeepSeek V4-Pro’s use Kimi.

• The panel wasn’t uniform, so read down a column rather than across a row. Opus and GPT-5.5 are different scales, not two measurements of the same thing. The one place I combine judges is the quality column in the listing, which averages only the two scorers that covered nearly the whole field.

The card below shows how far apart the lenient and strict judges landed on the same letters.

Kimi tops the Opus column and lands near the bottom of the strict one, and DeepSeek V4-Pro goes from 0.89 to 0.36 which is the widest gap in the table. Understanding why those judges saw the same letters so differently is part of this work.

The Results

This section covers the top scoring public model configurations, where the judges split on the same letters, how the models behaved, and what I took away. The full listing is at the end, with every judge’s score, cost per letter and median response time. It has 30 of the 35 configurations, and I explain what’s missing when we get there.

The short version first: the top eight of the 30, ranked on all four things I measured with quality counting for half. Quality here is the mean of Opus and Jev, the two scorers that covered nearly the whole field. The full listing at the end carries every judge’s column, so that’s where you can see where they split. Self-hosted rows show an estimated cost, what an H100 would run by the hour if it hadn’t been donated.

Both scorers behind that quality number are lenient ones, so this ranking sits on the lenient side of the split.

How the writer models behaved:

• The reasoning models cost a lot for what they gave back. Kimi-K3 used 3.3K tokens and 1 1/2 min / letter w/o tight prompt.GLM-5.3 used 6.3K and hit the 8,192-token response limit and returned no letter in 179 cases, and Qwen 3.8 27B on the H100 used 6.4K output tokens and over 5 min/ letter. A good follow up would be to run them with reasoning off to see if quality can hold up and improve cost.

• Most models write letters in markdown unless you tell them not to. Gemma, DeepSeek V4-Pro, GLM, Kimi, Opus and MiniMax put headings or bold in nearly every letter. Llama and GPT-5.5 mostly didn’t. Two prompt sentences fixed it.

• Size stopped predicting quality. Llama 3.2 3B averaged about the same as Llama 3.1 405B and Llama 4 Maverick on the lenient judges, and beat both on Jev.

• Families have habits, up to a point. Llama and Mistral sat lowest on citing the denial and arguing the case, on both judges, while Gemma 4, GPT-5.5 and DeepSeek V4-Pro engaged in nearly every letter. Llama 3.2 3B broke its own family’s pattern and scored with the top group. The head-to-head agreed: Llama letters lost to the reference side by side despite rubric scores at the field average.

• Perplexity Sonar was fast and cheap, about 6 sec, and mid-pack on quality. Where it falls behind is arguing the case: 0.78 on medical necessity against a field median of 0.85 on Opus. GPT-5.5 flagged about 11% of its letters, the lowest rate among the configurations it scored. Fewer flags, but a thinner argument.

• Our fine-tunes barely moved under the tight prompt, because they were already terse and never wrote markdown. That is what a fine-tune is for, and it’s one place they behaved better than the base models they came from.

• For the tight prompt exploration on just the Gemma models, it cut output tokens and response time by about 40%, and quality went up, nearly all of it on tone and form. Two sentences bought more than moving up a model size did because models pad by default which has clear costs.

What I learned about the judges:

• On the judges, GPT-5.5 flagged more letters than any other judge until I tried GPT-5.6, which agreed with 94% of GPT-5.5’s flags on the same letters and then called out more with 44% flagged against GPT-5.5’s 19% on one configuration. Even the two strict judges couldn’t agree on where the line is.

• Jev is the model I’d review more first on regular eval harness use. It ranked the writers the way the judge panel did and my read, it sided with the lenient judges, and it costs < $2 for all 21K letters and is 10x faster in scoring. I also need to revisit how I use the probabilities and confidence scores it provides.

My 20 Letter Judge Eval Side Quest: To decide which judge quality column to give more attention to, I read some letters myself. Claude pulled 20 from the Gemma 26B 4-bit tight-prompt configuration (10 that only GPT-5.5 had flagged and 10 that every judge had passed). Its not a large sample but it still had a signal because I marked 19 clean, including all 10 of GPT-5.5’s flags. The 20th was a handmade case with no real denial and it counts against the other judges that passed it. Opus, DeepSeek V4-Pro and Jev matched me on 19 of 20. Kimi matched 17. GPT-5.5 matched 9 and GPT-5.6 matched 7. The AI model judges also saw the review-board notes, which I did not have during my read, so some disagreements may come from that extra context.

So what was the strict GPT judge reacting to? Some of those flags were about what I consider good advocacy. I want a letter to bring in true knowledge about the condition, explain urgency and argue from the patient’s side, even when the point isn’t in the denial. Offering to attach records is fine. Picking an age inside the supplied range is fine, the patient can correct it. What I don’t want is fake references or invented facts about the denial, and unfortunately the judges didn’t know they should validate citations. They scored the letters without looking anything up. So the fabrication needs to be called out beyond just a gate and clarified that all sources and denial details should have a verified source. My read explained some false alarms but not whether the judges catch real fabrication.

Overall what I took away:

• The 4-bit Gemma 4 26B with the tight prompt is what came out of this. It scored 0.94 with Opus and 0.74 with GPT-5.5, second only to Sonar on the strict judge, at 3.8 seconds and 484 output tokens a letter. That is the fastest configuration in the field, it fits on one commercial GPU, and it was favored by the lenient judges and the strict one.

• Quantized builds held up generally: the 4-bit Gemma configurations scored the same as the full-precision ones and used less GPU memory.

• Several newer models outperform our fine-tuned models, which I expected. We already incorporated GPT-5.5 while our usage is covered by sponsored credit and added Gemma 4 26B to our external model line-up. And we retired one of our fine-tune models and have more reviews with the remaining.

• Make fabrication explicit esp. the rigor behind invented facts. The judges varied on the definition of invented facts and this needs more clarification next time.

• What the evaluation can’t show is whether an appeal wins, which is the whole point of writing one. It can help compare how models address a denial, how the judges score their letters, and what they take to run. Our fine-tuned models, including the one we retired, have helped people get their care covered, and all the models in this listing could do the same. What we’re after is lowering the barrier to filing an appeal at all, and improving what gets written and submitted.

The Cost

Something I’ve been interested in understanding better is what it all costs. We thought a lot about how to keep the model and infra costs down at FHI. It’s becoming all the more important with limited agent and compute access and increased expense. We want to know how much money and how many tokens to run things like this evaluation as well as the cost to write a letter (the latter is in the table at end). This section reviews the cost to run this evaluation setup which was about $4,250 without credit (~ $450 actual cash) and about 490M tokens.

I put together a table that shows what we paid and what it would have cost without sponsored credit. Two tables follow. The first goes line by line through what each judge and each group of writers used and cost. Cash is what we paid, reconstructed from recorded usage and rates where the bill wasn’t itemized. Sponsored credit is usage Azure and TypeSafe covered. The second chart rolls the same numbers up into writing versus judging groupings. The GPT-5.5 estimate uses recorded tokens from four configurations and extrapolates to earlier passes and its an example where judgements include the head-to-head comparisons like DeepSeek. GPT-5.6 uses GPT-5.5’s measured consumption as a proxy. The H100 estimate uses an illustrative hourly rate. Call counts are judgments on the kept rows, one for the rubric and one for the head-to-head, which itself ran as two requests.

The cash portion is about $450 and most of it judging. Azure sponsorship covered Opus, the GPT models, and the lovely H100. Four configurations ran on our own cluster rather than the H100, about $50 of GPU time at the same illustrative rate.

Some caveats to the numbers in the tables:

• For the H100 line, 156 hours and 48 minutes is elapsed time, not GPU-busy time or billed VM time. The machine served models for the eval and also sat waiting on API judges, and it was allocated before and after the logged window too, so the real number is would be higher. The $7/hr is a round number for renting one H100 by the hour, not what Azure charged us, since the machine was donated.

• The first attempts that were thrown away. GLM-5.3 returned 552 empty letters and Kimi hundreds of stubs because of a token cap that limited results. Those rows were regenerated with a higher cap, and the wasted calls are in the counts at about $15 on DeepInfra. GLM-5.3 still came back empty on 179 of its 632 letters at the higher cap, about $6 more, and those count as zeros in its scores.

• The smoke test. Before the real configuration runs, Gemma-3-27B wrote 16 letters across two tiny runs and Llama-3.3-70B judged them, to prove the harness worked. Those files are still in the repo, and it cost cents.

• The self-hosted per-letter prices are estimates, not invoices. Eight configurations ran on GPUs rather than an API, and the harness recorded $0.00 for most of them. I priced each one from its own generation time at ~$7/ hr, which aligns to the H100 line rate. Since letters were generated a few at a time, the real GPU-busy time is lower than the total I summed.

• And note there is one big gap in the numbers which are the agents who helped me set this up and run this aren’t counted in here.

Nine of every ten tokens were judging, and that breaks into three different things:

• Deliberate breadth. Opus and DeepSeek agreed on the fabrication flag for 99.7% of the Gemma 26B tight-prompt letters, but only about 75% of DeepSeek R1’s letters. Across the public configurations, both judges flagged only about a fifth of the letters either flagged. Human review is still needed to tell whether those disagreements are false alarms or missed errors.

• The head-to-head. Every letter with a reference got a second judgment, run as two requests with the order swapped. That’s the gap between 112,000 judgments and 165,000 API requests, and about half the judging. With one anchor for nearly every model it mostly told me who beats Gemma 3 27B, which wasn’t our best performer and isn’t something we ship. Directionally useful, not a ranking, so it stays out of the tables. Next time I’d anchor on what we run in production.

• Bugs. My rough estimate is that about 70M tokens went to retries and re-judging, the empty letters and the rows judged twice after the fix among them. The re-judged Opus rows are about $390 on top of the final rows.

For a smaller budgeting example, 100 cases across three configurations makes 300 letters. If two judges each score every letter once, that adds 600 scoring calls. At an illustrative average of under a cent to about seven cents per call, judging alone would cost $3 to $42. The actual amount depends on the judges, how much text they read, and how much they generate. Letter generation, retries, head-to-head checks, and any hosting costs are extra. I’d price a small pilot before scaling up.

What I’d Do Differently

It would be good to start with fewer models and calibrate the judges before scaling up. By calibration, I mean checking the judges’ decisions against human-reviewed examples and fixing the rubric where our meanings differ. I knew this going in and skipped it anyway. My 20-letter read helped expose false alarms, but it did not test how reliably the judges catch real fabrications.

With more time and a second reader, I’d aim for a deeper pass. I’d start with 150 letters, with two people independently reading all of them. I’d use 50 to work through disagreements and clarify the rubric, then freeze it and check the revised judges against 100 fresh letters. That second group would include 50 verified clean letters, including useful advocacy, and 50 containing verified errors, including deliberately planted ones. The readers would get the same case information as the judges, without seeing the model names, judge scores, or which errors were planted. I’d measure missed fabrications and false alarms separately. Citation verification would get its own step. And yes, I would actually check the citations. Whatever mistakes we find would become future test cases. With that foundation, I’d try more model judges and see which ones catch errors without rejecting the advocacy I want.

I own the evaluation harness now, including retries and incomplete runs, which is steps 3 to 5 in The Setup. An existing framework could take some of that off my hands. Inspect, Promptfoo, and Braintrust are the ones I’d look at for custom scoring and comparisons. For keeping runs and judge scores in one place with a review interface, Arize Phoenix and W&B Weave would make it easier to look at disagreements together. Whichever tool I use, I still need examples of what I mean by a good appeal, and no tool would have caught the citation verification issue for me. For a small version, I’d start with a laptop, Python, API access, and a few dozen cases. Running model weights adds hardware and serving work. I want to experiment with some of these tools and hope to at some point soon.

Now What

I started this to evaluate our own fine-tuned models, and even though their numbers stay with us, we did learn a lot including all the points about the public and frontier models here. We now know which models we want to keep using and which can retire. It’s an open question on whether we fine-tune more because we want to know what it should improve over the models already available, what needs to change in the training data, and how I’d tell whether those changes helped. Are there newer models we should use? Yes. I found promising newer models for this task, and smaller models and tighter prompts played a big role in choosing the 4-bit Gemma 4 26B as a model to use going forward.

If anything this experience has given me more questions to explore. Could fewer judges give the information we need? Can we get good results out of a smaller evaluation dataset? When do head-to-head comparisons change the decision? How much can the judges catch real fabrication without rejecting useful advocacy? Do we want to use tools to help optimize evaluations?

This was a fun one to dive into. Now, I can take a question about a model or a prompt, run the comparison, and open the letters behind the result. Saving every output separately paid off once the judges and approach became the thing I wanted to investigate more. We much more solid ground to work from than where we were at so the next model decision gets made against structure, not a feeling. Working with the agents made it practical in the time I had, and it was fun to explore and get to this point.

The Listing

Every configuration except our fine-tunes, ranked on all four things I measured: quality counts for half, then speed, output length and cost. Our fine-tunes were trained on data from the same state records this set came from and we are doing further analysis.

Quality is the mean of the two scorers that covered nearly the whole field, Opus and Jev. A letter a judge flagged for invented facts counts as zero in that judge’s column, so a configuration’s number is pulled down by every flagged letter rather than wiped out. A dash means that judge never scored that configuration. Seconds are the median per letter and depend on how the model was served that day. Tokens are output only, and tokens and cost are means per letter. The self-hosted rows are priced from their own generation time at the same illustrative $7/hr as the H100 line in The Cost, so they are a share of that line rather than extra spend.