Choose Models with Your Own Work, Not a Leaderboard

Ask two questions about each task

I can burn through my allowance by keeping the strongest model on a job long after the hard decisions are made. I now ask: how much judgment does this step require, and how quickly can I check it? Direction and trade-offs are harder to assess than extracting fixed fields or changing a format.

When a task looks simple but has no clear acceptance criteria, I define those first. A lighter model can otherwise produce more work for me to review.

Make a three-question work test

Choose anonymized samples: a weekly task, a task where invented information would hurt, and a task with strict formatting. Keep the task, prompt, input sample, expected output and forbidden behavior fixed.

• Meeting notes: missing owners and dates must remain unresolved.
• Data extraction: each number has a source; missing values stay missing.
• Structured output: all required fields appear in the agreed order.

Hide model names before judging

My comparison method runs each model twice per task, removes the model names and shuffles the four outputs. I check correctness, format compliance and invented information before revealing the identities.

This small work test helps me locate recurring failures. AI can point out differences; whether the writing fits and the trade-offs make sense remains my judgment.

Keep limits, money and rework separate

I have calculated what a recorded task would cost at API rates. That was a hypothetical bill, not a credit-card charge. Subscription allowance and API spending need separate records; a money calculation needs prices checked on the calculation date.

I would record allowance changes, retries and review time for the same batch. Missing measurements stay marked “unavailable.” Local token output or a percentage of a monthly fee cannot establish money saved.

Change only the task that benefits

Try one low-judgment, easy-to-check task for a week. Expand only after quality and rework are acceptable. A model winning one task is a reason to route that task differently.

The decision is which arrangement gives you a trustworthy result for this job. Different jobs can keep different arrangements.