Many conversations with clients this year come to the same question: do we really need OpenAI, Anthropic or Google for this, or can we run an open model ourselves? Often it comes with a vendor leaderboard on open-source vs commercial LLM accuracy attached, which measures something else.
I run 4xxi, and part of my time I am an astronomer: in 2026 Valerio Carruba and I published a benchmark in Scientific Reports that answers a narrow version of this question with real numbers. We gave the same expert classification task to 16 multimodal models, from GPT-5 and Gemini 2.5 Pro to Gemma 3 4B, and compared every answer with expert labels. Below is what I take from it for product work, and where I would not stretch it.
Short answer. In our peer-reviewed benchmark of 16 multimodal LLMs (Smirnov & Carruba, Scientific Reports, 2026), the best open-source models were as accurate as closed commercial ones on clear cases: both groups reached F1 = 100%. On a binary set of 450 images, the open models we ran scored 89–93% against 93–96% for commercial ones. The gap opened on ambiguous cases: 94% for Gemini 2.5 Pro, 76% for the best open model, Llama 3.2 90B. So the question to ask first is how much of your traffic is hard.
What exactly did we test?
We tested one thing: whether a multimodal model can look at a plot and put it into the right class. The plot shows how the resonant angle of an asteroid evolves over time. An astronomer looks at it and says ‘librates’ (the asteroid is trapped in a resonance), ‘circulates’ (it is not) or ‘something in between’. For clean cases, simple algorithms do this well. For the messy ones in crowded regions of the main belt, visual inspection by an expert has been the only reliable option, and there can be hundreds of candidate resonances per asteroid. (Earlier I attacked the same problem with classical machine learning.)
The paper is the second step. In 2024 I showed in The Astrophysical Journal that one model, gpt-4-vision-preview, could classify these plots with 100% accuracy on simple examples, without training or fine-tuning. That answered ‘can it work at all?’. The 2026 study asked how well it works across vendors, and on hard cases.
We built four datasets:
- RB-TEST: 8 clear images, a quick sanity check.
- RB-PILOT: 50 images, two classes (resonant / non-resonant).
- RB-SMALL: 50 images, three classes (resonant / non-resonant / transient), with controversial cases on purpose.
- RB-FULL: 450 images covering every behaviour in our taxonomy.
The models fell into three groups: commercial (Gemini 2.5 Pro, Flash and Flash-Lite, GPT-5 with mini and nano, Claude 4.5 Sonnet and Haiku, Grok 4 Fast), large open-source (Llama 4 Maverick, Llama 3.2 90B, Mistral Medium 3.1) and small open-source (Gemma 3 12B and 4B, Qwen3-VL 8B, Mistral 3.2 Small). Images went through each vendor’s API, OpenRouter or a local interface; answers came back as JSON. No training, no fine-tuning: only a prompt. The datasets and the code are public.
How accurate are open-source LLMs compared with closed models like GPT, Claude and Gemini?
On easy data there is no gap: the best open and commercial models both reached F1 = 100%. On hard, ambiguous data there is a big one: 94% for the best commercial model, 76% for the best open one. Here are F1 scores for a representative subset of models from Table 2 of the paper (‘n/r’ means the model was not run on that set):
| Model | Group | RB-TEST (8, clear) | RB-PILOT (50, binary) | RB-SMALL (50, 3 classes) | RB-FULL binary (450) |
|---|---|---|---|---|---|
| Gemini 2.5 Pro | Commercial | 100% | 100% | 94% | n/r |
| Claude 4.5 Sonnet | Commercial | 100% | 100% | 89% | 93% |
| GPT-5 | Commercial | 100% | 100% | 79% | n/r |
| GPT-5 mini | Commercial | 100% | 99% | 89% | 96% |
| Gemini 2.5 Flash | Commercial | 100% | 99% | 81% | 94% |
| Llama 3.2 90B | Open, large | 100% | 94% | 76% | 93% |
| Mistral Medium 3.1 | Open, large | 100% | 97% | 64% | n/r |
| Gemma 3 12B | Open, small | 100% | 94% | 52% | 89% |
| Gemma 3 4B | Open, small | 77% | 80% | 67% | n/r |
| Qwen3-VL 8B | Open, small | 77% | 85% | 56% | n/r |
On RB-TEST, 10 of the 16 models were perfect. On RB-PILOT, the three flagships (Gemini 2.5 Pro, Claude 4.5 Sonnet, GPT-5) scored 100%, and the best open models were close behind. On RB-SMALL, everything dropped: the open models other than Llama 3.2 90B ranged from 52% to 67%. On the hardest setting, all 450 images with three classes, we only ran three commercial models, and the best of them reached about 75%.
A caveat: the research had no funding, so the 450-image runs were limited to the models that did best on the smaller sets. The empty cells are a budget decision, not a result.
Where do the models fail?
They fail where an expert slows down too. Most errors sit in transient behaviour and resonance sticking: an asteroid that is trapped for a while, escapes, or wanders near the boundary. On RB-FULL, most misclassifications were circulation with sticking and chaotic behaviour with short episodes of libration or circulation. These are exactly the cases where, as we note in the paper, agreement between human experts also goes down.
The models do not fail randomly. Moreover, different models made errors on different images, which suggests that an ensemble of mid-tier models might reach flagship accuracy at a lower price. We did not test this; it is a hypothesis.
What does it mean for a business task? Your hard cases are the ones where your own staff would argue: a clause that might be a termination clause, a transaction that is unusual but not obviously wrong. If such cases are 2% of your traffic, model choice hardly matters. If they are 30%, it is the whole decision.
Is a bigger model always better?
No. Three observations from the same table surprised me.
- GPT-5 mini beat GPT-5 on the hard set: 89% against 79% on RB-SMALL. The larger model was not the better classifier.
- A good score on easy data predicts little. Gemma 3 12B scored 94% on RB-PILOT and 52% on RB-SMALL. If we had stopped at the pilot set, we would have recommended it for the wrong job.
- F1 can hide a useless model. Gemma 3 4B scored F1 = 80% on RB-PILOT by answering ‘resonant’ for every one of the 50 images. Perfect recall, zero true negatives. Gemma 3 12B on RB-SMALL did the opposite: 100% precision, 35% recall. It never raised a false alarm, and it missed about two thirds of the cases it should have caught.
[T]he optimal model for a given classification task may not simply be the largest or most sophisticated available and therefore has to be studied in advance by researchers.”
How far do these results transfer to your product?
The numbers do not transfer; the pattern does.
This is one task: images of time series in, one of two or three labels out, zero-shot, on datasets labelled by two astronomers (Valerio confirmed my manual classification). The model versions are those available in 2025 (the paper was submitted that November), and several already have successors. Your task has a different input, a different definition of ‘hard’ and different costs of error. Hence, nobody should read ‘76% for the best open model’ as a property of open models in general.
What I do expect to carry over to business classification tasks (triaging documents, tagging tickets, sorting scanned forms, flagging items for review):
- Clear cases are cheap. Most modern models handle them; paying flagship prices for them is waste.
- The dataset decides the score. In the paper we argue that comparing results across different datasets is incorrect: on a naturally imbalanced sample, where easy cases dominate, even basic open models would likely look very good. Vendor benchmarks and your traffic are different datasets.
- Size is not monotonic. The GPT-5 / GPT-5 mini result is a warning against choosing by price tier.
- The prompt has to match the model. More on this below.
Interestingly, we saw no systematic difference between mean-motion and secular resonances, two physically different phenomena: the models seem to react to the shape of the plot, not to the physics behind it. Within one domain, a well-defined task seems to generalise. Across domains, I would test, not assume.
Can a small local model do real work?
Yes, if you give it a task its size can carry. Gemma 3 12B reached F1 = 89% on our 450-image binary set, but only 52% on the 50-image set built around ambiguous cases. This part of the study most resembles our client work: the small models did not behave until we changed how we instructed them.
For commercial and large open models we wrote a full prompt with an explicit algorithm: split the image into blocks of 10%, classify each block, combine the results into three classes. Pilot runs showed that small models could not follow it, so we wrote a simplified prompt: two classes instead of three, plain language instead of terminology, only features one can see in the plot, and hard rules in capitals (‘the status is ALWAYS resonant’, ‘review the WHOLE graph’). The opposite also held: large models did worse with the simplified prompt, which drops the logic that helps them on ambiguous cases. When we tuned the prompt for one specific model, that model improved and the others got worse.
The lessons we apply when instructing small models for a specific task:
- Shrink the task before you shrink the model. Fewer classes, binary decisions, or a cascade: the paper suggests a two-stage approach, resonant versus non-resonant first, transient second.
- Describe what is observable, not the theory. A small model follows ‘parallel diagonal lines across the entire plot’ better than a definition.
- Bias the prompt towards the error you can afford. Our simplified prompt sends any plot with oscillations, even partial ones, to ‘resonant’. The study’s three-class scheme favours recall, because in a population survey missing an interesting object costs more than checking a false one.
- One prompt per capability tier, not per model, unless you are committing to that model. Per-model tuning makes comparisons meaningless.
- Re-run the golden set on every model swap. A new version of the same model is a new model.
In the paper we only changed prompts: no training, no fine-tuning. In client work, where prompting is not enough, we fine-tune. Two examples, both small models that run on the client’s own hardware:
- Anonymising legal documents. A small model that removes personal and client details from legal documents, so that nothing extra is left in them before they go anywhere else. We fine-tuned it on documents the client provided, and it does this very precisely, locally on an ordinary computer or on the client’s own server. The documents never leave their network to be cleaned.
- Finding citations in documents. A local model that detects citations and references in text. We fine-tuned it for two reasons: to cut the cost of sending every document to a commercial API, and, again, privacy.
The pattern is the same in both: a narrow task with a clear output, enough examples from the client’s own documents, and a good reason for the data to stay local. If one of these is missing, I would start with prompting, as in the paper, because a lot can be done before any training at all. The general trade-off is in RAG vs fine-tuning.
Is a private LLM on your own servers good enough, and what does it cost?
On our benchmark, yes. If your data cannot leave your network, an open model on your own infrastructure is the only option, and it is not a toy one: Llama 3.2 90B reached 93% on the 450-image binary set, the same as Claude 4.5 Sonnet, and Gemma 3 12B, which runs on a standard workstation, reached 89%.
I made the same argument for researchers at my ECP 2026 keynote: open families such as GLM, Kimi, MiniMax, DeepSeek, Mistral, Qwen, Llama and Gemma are usable, and running them locally means sensitive participant data never leaves your own infrastructure. For a legal publisher or a bank the logic is the same.
We built this into our own product. In Edtek, the model is a setting per chatbot instance: OpenAI, Azure OpenAI, OpenRouter or any OpenAI-compatible endpoint, and the whole stack can run on-premise so that content and inference can stay inside the customer’s network. The same question comes up whenever we add an LLM to software you already run.
On cost, the paper is plain: flagship models need paid API access, which gets expensive over thousands of images, while small models run on ordinary hardware. I won’t give a cost-per-answer figure: it depends on your volume, your prompts and this month’s price list. Here is how we estimate it for a specific feature. One might, however, consider a hybrid: a small local model for the clear majority and a flagship for the cases it flags as uncertain. It follows from the error pattern above, but measure it on your data first.
How do you run the same kind of benchmark on your own task?
You need a few dozen labelled cases and a small script, not a research grant. The checklist we follow:
- Start with around 50 real cases. Our RB-PILOT and RB-SMALL were 50 images each: enough to separate models, small enough to label by hand. Bear in mind that on small sets one error moves the percentages a lot.
- Over-sample the hard cases. Include ambiguous examples at a higher rate than they occur in production, and write down what ‘hard’ means for you.
- Label twice. Two people label independently; the cases where they disagree are your ‘transient’ class.
- Fix the prompt per tier and ask for structured output (JSON), so parsing errors do not pollute the results.
- Run several models through one gateway (OpenRouter, or Ollama for local models; our code supports both), with identical inputs.
- Report the confusion matrix, not only F1. Look for the Gemma 3 4B pattern: a decent score from a model that always answers the same thing.
- Keep the set. It becomes your regression suite for the next model release.
This is the same discipline I describe in scientific rigour in AI development, and it is one of the first things we check when we audit an existing AI system.
When is the flagship model worth paying for?
When the hard cases are a meaningful share of your traffic and an error on them is expensive. On RB-SMALL the difference between the best commercial and the best open model was 18 points, and that is the kind of gap you pay for.
Furthermore, I would choose a flagship when:
- the task needs long, multi-step instructions (exactly what small models could not follow in our study);
- you are at the prototype stage and volumes are low, so the API bill is usually small compared with engineering time. Start from the ceiling, then try to go down;
- your data may leave the network and nobody wants to operate model infrastructure.
Commercial, open-source or local: which should you choose?
Choose by the share of hard cases and by where your data may go, not by price tier. On one task, this is what our benchmark suggests:
| Your situation | What the benchmark suggests |
|---|---|
| Mostly clear cases, data may leave your network | Most current models will do; 10 of 16 were perfect on our clear set |
| Data must stay inside your network | A large open model on your own servers (Llama 3.2 90B: 93% on the 450-image binary set, the same as Claude 4.5 Sonnet) |
| Limited hardware, narrow task | A small open model with a simplified two-class prompt (Gemma 3 12B: 89% on the binary set, 52% on the ambiguous one); fine-tune it on your own examples if prompting is not enough |
| Many ambiguous cases, expensive errors | A flagship model (Gemini 2.5 Pro: 94% on the hard three-class set, against 76% for the best open model) |
| Mixed traffic | A small model for clear cases and a flagship for the ones it flags: a hypothesis, so measure it on your data first |
When can you skip all of this?
When model choice is not your bottleneck:
- Your cases are clear and volumes are low. Pick a decent model and move on; on clear cases, most models reach the ceiling.
- Nobody can say what a correct answer is. Then there is nothing to benchmark yet. Define the task and collect labelled examples first.
- Your problem is answering from documents. There, retrieval and grounding usually matter more than the model; see how to build an assistant that doesn’t hallucinate.
- You have an in-house team with time for it. The checklist above, our code and our datasets are public. You can do it without us.
Which model does your task actually need?
If you want the answer measured on your own cases rather than on a vendor leaderboard, start with our AI opportunity audit: 2 weeks, $5–10k fixed, full money-back guarantee, and the fee is credited toward any project signed within 60 days. Model choice (commercial, open or local) is one of the questions it answers, and if you already have an AI feature, we review it too. If you already know the task and want to see it working, a prototype sprint gives you a deployable prototype on your data in 5 business days for $20k fixed, and it can include the evaluation set from the checklist above. Or book a call and bring your hardest examples.
My takeaway after two papers: flagship models are excellent but not the only option, and the benchmark that matters for you is the one built from your hard cases.
P.S. The full paper is open access: DOI: 10.1038/s41598-026-45926-y. If you want to test a model that did not exist when we ran the study, the datasets are on Zenodo. I wrote up the astronomy side on my blog.