Most teams evaluating multimodal AI for document work do it the same way. They try a handful of sample files, glance at a public leaderboard, and pick the model with the best reputation. Then they find out in production that the model is great at the easy parts of their documents and unreliable on the parts that matter.
We wanted a harder test than that, so we built one on a task where “almost right” is visibly wrong: finding objects on architectural drawings. The result is CaseV-Bench, an AI architectural drawing benchmark that we re-run whenever a notable model is released. The September run covered 11 models from OpenAI, Anthropic, Google, Alibaba and xAI. The findings are specific to construction drawings, but the lessons apply to any team choosing an AI model for document work.
Also Read: AI-Powered vs. Traditional SIEM: What Should Enterprises Choose in 2026?
The Test
We took 119 pages from nine publicly available drawing sets. An annotator marked every object of five types by hand: cabinets, countertops, floor plans, interior elevations and callouts. That came to 1,430 objects, and it serves as the answer key.
Every model got the same pages, one page at a time as a single image, with the same prompt: find these five types of objects and return a box around each one. A scoring script then compared the model’s boxes with the annotator’s.
A box only counts if it overlaps the annotator’s box closely enough. The rule is intersection over union (IoU) of at least 0.50: the overlapping area must be at least half of the total area covered by the two boxes. From the correct, stray, and missed boxes, we compute F1, a single score that only rises when a model finds more real objects and invents fewer fake ones.
The Results
| Model | All objects | Callouts | Countertops | Cost per page | Time per page |
|---|---|---|---|---|---|
| GPT-6 Astra | 92% | 96% | 84% | $0.220 | 39 s |
| Claude Fable 5.1 | 70% | 64% | 34% | $0.164 | 26 s |
| Gemini 3.8 Flash | 70% | 68% | 20% | $0.074* | 104 s |
| GPT-5.6 Sol | 65% | 57% | 24% | $0.060* | 49 s |
| Qwen3.8-Max | 62% | 47% | 32% | $0.057 | 161 s |
| GPT-5.6 Terra | 56% | 38% | 23% | $0.056 | 32 s |
| Gemini 3.5 Flash | 53% | 33% | 16% | $0.035 | 20 s |
| Gemini 3.1 Pro Preview | 51% | 37% | 10% | $0.040 | 23 s |
| Claude Opus 5 | 40% | 15% | 19% | $0.134 | 48 s |
| Grok 4.6 | 30% | 0% | 7% | $0.145 | 362 s |
| Claude Sonnet 5 | 21% | 4% | 0% | $0.073 | 60 s |
F1 at IoU 0.50. *Billed at a 50% promotional rate during the run; at list price, roughly $0.12 (Sol) and $0.15 (Gemini 3.8 Flash) per page.
GPT-6 Astra is far ahead at 92%. Claude Fable 5.1 and Gemini 3.8 Flash share second place at 70%, and nothing else clears 65%. Five lessons stand out.
1. Leaderboards go Stale in Weeks
When we ran the first six models in August, the top score was 62% and the leaders were bunched together. A single release moved the top of the table by 30 points.
Upgrades within a model family were just as large. Gemini 3.8 Flash beats Gemini 3.5 Flash by 17 points. Claude Fable 5.1 beats Claude Opus 5 by 30. GPT-6 Astra beats GPT-5.6 Sol by 27. If your evaluation of a model is more than a couple of months old, it probably no longer describes the models available today.
2. The Easy Parts Hide the Hard Parts
Floor plans are large, often a quarter of the sheet. Every model except Claude Sonnet 5 found at least 89% of them. Judged on floor plans alone, nearly any model looks production-ready.
The ranking is decided on the small objects. A callout can be a few pixels wide once a full sheet is shrunk to the image size a model accepts. A countertop is a thin line sitting on top of a cabinet run. On countertops, no model except Astra scored above 34%. On callouts, the spread runs from 96% down to 0%.
Most business documents have the same split: a few big, obvious elements and many small, dense ones. A demo tends to show the first kind. Your error rate comes from the second.
3. Look at How a Model Fails, not Just How Often
A single accuracy number hides very different behavior. On one plan sheet with a single callout, Qwen3.8-Max drew 67 callout boxes. It did find the real callout, so its recall on that page was 100%. But 66 of its 68 boxes on the page pointed at nothing, and its F1 was 6%. This is why we score with F1 rather than “percentage of objects found.”
Direction matters too. Astra almost never misses an object: it finds 95% of them, and most of its errors are extra boxes. Other models skip things rather than guess. For the person checking the output, those are different jobs. Deleting extra boxes is quick. Hunting for a missing cabinet on a dense sheet is slow.
Box quality is its own signal. When we tighten the overlap requirement from 0.20 to 0.50 to 0.70, Astra’s score goes from 93% to 92% to 82%. Claude Fable 5.1 goes from 85% to 70% to 43%. Both models see most of the objects, but only one draws boxes a person can trust without redrawing them.
4. Price and Speed don’t Track Accuracy
At the top, you do get what you pay for: Astra is the most expensive model in the set at $0.22 a page, and the best by a wide margin. Below the top, the link breaks. Claude Opus 5 and Grok 4.6 cost roughly three to four times as much per page as the older Gemini models and score lower.
Speed varies even more than price. Per-page time runs from 20 seconds to 362 seconds, an 18-fold spread, and it doesn’t follow either cost or accuracy. For a 100-page drawing set run page by page, Astra costs about $22 and takes about an hour. A slow model can tie up most of a working day on the same set.
5. Your Documents are the Only Benchmark that Counts
The same model can score very differently on different drawing sets. Astra ranged from 73% to 100% across our nine projects. In our August run, Qwen3.8-Max ranged from 44% to 88%. Density, drafting style, and how cleanly a set was produced move the score as much as the choice of model does.
This means a vendor quoting a single accuracy figure is referring to one set of documents, and those documents probably aren’t yours.
Also Read: How to Build an AI Governance Framework for Enterprise IT in 2026
What This Means When You Choose a Model
A few practical rules follow from this.
Test on your own documents, including the dense and messy ones, not only the clean samples a vendor sends.
Score the small, high-value elements separately. An average dominated by easy objects will overstate the tool’s usefulness.
Measure review time as well as accuracy. A model that over-detects and one that under-detects can have similar scores and very different cleanup costs.
Re-run your evaluation when new models ship. In this category, a few months is a long time.
Look at the system, not just the model. Each result above is produced by feeding a model a full page as a single image. Cropping the relevant views and analyzing them at higher resolution substantially raises the scores of the weaker models. For many document workflows, the pipeline around the model matters as much as the model itself.
Final Thoughts
Finally, keep the scope honest. Our benchmark measures one step: locating objects on a page. A real construction takeoff also needs dimensions, schedules, and an estimator’s judgment. On the step we measured, the best model now gets roughly 9 out of 10 boxes right, which makes it a useful first pass for a person to check. It still doesn’t replace the person.
