We pitted an LLM against Cloudflare's Clairic on a pile of monsters. Here's what we learned
A Halloween workshop, a head-to-head extraction test, and a follow-up experiment with sharper field instructions. Here's a practical guide to choosing between a general-purpose LLM and a purpose-built extraction model.
Most teams that need structured data from messy text or images reach for an LLM, ask for JSON, and move on. It works well enough that few people ask whether it's the right tool. Three days after Cloudflare released Clairic, a purpose-built extraction model in preview on Workers AI, we ran a workshop to find out.
Mixed teams of designers and engineers spent three hours extracting monster facts from spooky text and images, twice: once with Claude Sonnet 5.5 and once with Clairic. Same schema, same data, same scoring. The monsters were a theme. The question was real: when does each approach win?
The short version
On our Round 2 data (20 images):
| Claude Sonnet 5.5 | Clairic: workshop baseline | Clairic: follow-up | |
|---|---|---|---|
| Field accuracy | 83% | 59% | 63% |
| Weaknesses F1 | 95% | 40% | 60% |
| Median time per item | 2.4 s | 0.78 s | 0.76 s |
| Cost per 1,000 items | $6.60 | $0.62 | $0.63 |
| Responses that parsed with correct types | 20 of 20 | 20 of 20 | 20 of 20 |
The follow-up combined a sharper instruction for the weaknesses field with span-based post-processing. It helped on images, lifting Clairic's overall field accuracy by three points and its weaknesses F1 from 40% to 60%. But a parallel text experiment went the other way: weaknesses F1 fell from 12% to 9%. That reversal is as useful as the improvement.
The LLM was more accurate. Clairic was about 3x faster and about 10x cheaper. Neither result is surprising on its own. The interesting part is where the accuracy gap came from, and why the same tuning helped one workload and hurt another.
How we set it up
Every team extracted the same six fields from every item:
| Field | Type | Scored as |
|---|---|---|
| name | text | exact match |
| creature type | text | match against a synonym list |
| weaknesses | list | F1 over the items |
| size (metres) | number | correct within 10% |
| first recorded year | number | exact |
| danger rating (1 to 10) | number | correct within 1 |
Round 1 was 20 stat blocks and folklore passages from the open D&D SRD. Round 2 was 10 images: ten public-domain museum pieces (Goya's Caprichos, gargoyles, demon prints from the Met's open collection) and ten "Wanted" posters. Each poster had one deliberate curse, such as a rotated page, a decoy monster in the background or a misleading caption.
A judge held the answer key. We logged latency, token counts and the raw response for every item, so any score could be traced back to what the model said.
After the workshop we kept the original results and made an opt-in change to the Clairic pipeline: a more specific weaknesses instruction, followed by a filter that checks returned values against their source spans. We ran both 20-item datasets again with that change. The original LLM and Clairic numbers below are preserved separately, rather than silently replaced by the follow-up.
What Clairic is, and why it's different
An LLM generates its answer one token at a time. You ask for JSON, parse it and hope the types hold. Clairic works differently. You give it the source and a map of named, typed fields, and it returns values for all of them in a single parallel pass. Text fields come back as spans of the input. Numbers come back as numbers. Lists come back as lists.
Two practical consequences:
- Output is never malformed. Both models parsed cleanly in our run, but only one of them guarantees it.
- Pricing is input-only. Cloudflare quotes $0.10 per million input tokens and no charge for output. The LLM charged $2 per million in and $10 per million out.
Cloudflare also claims a 49% cheaper, 3.2x faster pipeline against an all-LLM version of a claims workflow. We wanted to see whether anything like that held on a different task.
Finding 1: speed and cost gaps were large
Clairic's workshop median response was 775 ms against 2.4 s for Sonnet, and its slowest item took 1.3 s. The follow-up median was 758 ms, so the local post-processing didn't change the practical latency story. In the two Sonnet runs we kept, the slowest items took 3.1 s and 8.5 s. The speed advantage is consistent with Cloudflare's own claim.
On cost, Clairic read more input tokens than Sonnet did across the 20 images (about 123,000 against 59,000), but it charges a fraction of the rate and nothing for output. That came to roughly $0.62 per 1,000 items against $6.60. If you extract a few hundred documents a month, this is noise. If you extract millions, it's a budget line.
Finding 2: accuracy depended on what was on the page
This is the part worth reading twice.
| Round 2 subset | Sonnet | Clairic: baseline | Clairic: follow-up |
|---|---|---|---|
| Wanted posters (printed facts, 10 items) | 100% | 83% | 87% |
| Museum art (facts mostly implied, 10 items) | 67% | 35% | 38% |
On posters, where the facts are printed in the image, both models did well. Sonnet got every field right on all ten. Clairic got most of them, with errors like "Jenny Greente" for Jenny Greenteeth and "Mother Mooth" for Mother Moth. The follow-up didn't change those two examples, but it did improve the overall poster score from 83% to 87%.
On museum art the gap opened up. These are etchings and engravings with a title and a date, and not much else. The facts a schema wants, such as size, weaknesses and danger, mostly aren't there. That tests a model's ability to say "the source doesn't say."
Finding 3: "I don't know" is a feature
Across Round 2 there were 33 fields (not counting danger rating, which is always an estimate) where the source simply didn't state the answer.
- Sonnet left 26 of them blank.
- Clairic's workshop run left 2 of them blank and filled in the other 31.
- The follow-up left 4 blank and filled in 29. Better, but not a solution.
In practice, Clairic invented sizes for all ten museum pieces. The baseline gave a vampire etching a size of 20 metres. It also returned single letters as names ("G", "g") and "s" as the creature type for a sphinx. Our changes touched only weaknesses, so those problems remained.
Sonnet's mistakes were of a different kind. It left danger rating blank on all ten museum images. It also gave a confidently wrong answer in one case: the etching titled "The Vampire" came back as a gargoyle, which contradicted the title right next to it. And it wasn't perfectly stable. Two runs of the same model on the same 20 items produced identical output for 15 of them.
Neither model "hallucinated" in the dramatic sense. Both filled gaps in their own way, and which you'd prefer depends on whether a missing value or a wrong value costs you more.
Finding 4: better instructions helped on images, but not on text
The weaknesses field exposed a lesson that applies to any structured extraction. Clairic's list items are meant to come from the source, but our starting instruction, "everything that harms, repels or defeats it", had no boundary. Stat blocks describe weaknesses three ways: a named section, a negated immunity such as "nonmagical attacks not made with silvered weapons", or a procedure such as "unless it has taken fire damage". A vague instruction can pull in clauses instead of terms, and one wrong clause costs precision even when the right item sits beside it.
We tested three changes together:
- Sharpen the field instruction. Say what counts, what doesn't and that you want short terms.
- Post-process with the spans. When text answers include character offsets, slice the source at those offsets. Drop items that don't match or that start like a clause ("if", "when", "unless"). If the number of offsets doesn't match the number of values, don't guess at the alignment.
- Use confidence to decide where to look. On our sample monster, weaknesses came back at about 0.38 confidence, against about 1.0 for the name. In the follow-up, most museum weakness lists had low confidence while the poster lists were mostly above 0.8. That makes confidence a useful review cue, not an automatic correctness score.
On images, the results were encouraging. The sharper instruction split "iron, running water" into two terms and completed "being pulled from the" as "being pulled from the water". Span filtering also removed two unsupported museum weaknesses. Across all 20 images, weakness precision rose from 27% to 48% and recall from 54% to 77%. The mean per-item F1 rose from 40% to 60%.
On text, the same approach didn't generalise. A fresh baseline and follow-up produced these results:
| Round 1: 20 text items | Baseline | Follow-up |
|---|---|---|
| Mean weaknesses F1 | 12% | 9% |
| Weakness precision | 7.6% | 5.7% |
| Weakness recall | 48% | 44% |
| Overall field accuracy | 65% | 63% |
Both text runs were poor at weaknesses. The follow-up returned more unsupported fragments, including headings and single letters, and dropped some valid terms. On a vampire, for example, the follow-up still returned running water and Stake to the Heart, but alongside noise such as M, V and L. The exact-match scorer accepted sunlight but not the capitalised stake phrase, while every extra fragment reduced precision. A perfect span check cannot rescue a list whose contents are wrong. The low absolute scores also show why a three-point gain on one field shouldn't be oversold.
There was another implementation lesson in the raw responses. Some text answers contained more spans than values. Pairing them by array position would validate the wrong slices. We added a guard that leaves such lists unchanged, and tests for that case. Image answers had a different edge case: their spans were often null, because there were no offsets into the accompanying text. Those values couldn't be span-checked and were retained.
So the useful conclusion isn't "always add this filter." It's "test field instructions and post-processing separately on each input type." Our combined follow-up doesn't tell us how much of the image gain came from the instruction rather than the filter.
So, which one?
Our numbers point to a split rather than a winner.
| If your work looks like... | Lean toward |
|---|---|
| High volume, tight latency or cost, and fields that are usually stated in the source | Clairic |
| Printed or typed content where the facts are on the page | Either; Clairic is cheaper and faster |
| Documents where many fields are often absent, and a blank beats a guess | LLM, or Clairic with a confidence gate |
| Ambiguous, narrative or judgement-heavy content | LLM |
| A hard requirement that output is always well-typed | Clairic |
A hybrid is the most interesting option. Run Clairic first on everything. Send only the low-confidence or high-null-probability items to an LLM. You keep most of the speed and cost advantage and reserve the expensive model for the cases where it earns its price.
What we'd tell developers
- Define the schema first. The six-field schema did more work than any prompt. It made the two models comparable and gave us something to score.
- Score per field, not per document. Our overall numbers hide the real story: Name accuracy, weakness F1 and size accuracy moved in completely different ways.
- Keep the raw responses. Every interesting finding came from reading actual outputs, not from the summary score.
- Test what's absent. Include items where the source doesn't contain a field. How a model behaves on missing data is often what separates two candidates.
- Run every model more than once. If you're going to put it in production, measure its variance. Two LLM runs weren't identical, and our follow-up Clairic runs also differed, so treat small score changes cautiously.
- Test prompt changes on text and images separately. Our weaknesses instruction improved image extraction and made text extraction worse. A single aggregate score would have hidden that.
- Validate your validator. Span counts, missing offsets and duplicate values are real edge cases. Keep raw responses so post-processing bugs don't masquerade as model failures.
What purpose-built extraction models could mean for you
As a business, the prize is structural. If extraction stays ten times cheaper and three times faster than a general model, workloads that were batch jobs become interactive, and workloads that were too expensive to run at all become viable. More importantly, a model whose output is always typed reduces the operational tax of validating and repairing malformed data, which is where most of the hidden cost in LLM pipelines actually lives. The catch, as our monsters showed, is governance: a cheaper model that fills gaps confidently is still expensive in review time. Purpose-built extraction changes your unit economics, but only once you have per-field scoring and a routing policy for low-confidence cases.
As a developer, the shift is from prompt engineering to schema engineering. Your leverage moves out of the prompt and into field definitions, per-field instructions and post-processing. Typed output with spans and confidences gives you handles a raw LLM response doesn't: you can verify a value against its source, filter list items, and route low-confidence items to a second model, all in ordinary code. The design space opens up, with the extraction model first and the LLM as the exception handler, but it rewards measurement discipline. Teams that score field by field and keep raw responses will tune these models well; teams that eyeball a demo will ship a gap-filler.
Caveats, because we're not publishing a benchmark
This was a three-hour workshop plus a small follow-up, not a lab. Each dataset has 20 items, and most comparisons are single runs. We used Sonnet 5.5 at low effort so a round would fit in the time box, and a different setting would change the result. Clairic is in preview, and Cloudflare says so. Our scoring rules, such as exact-match names and F1 over weaknesses, are strict on purpose and would punish near-misses a human reader would accept.
The follow-up changed the weaknesses instruction and enabled span filtering together, so it doesn't isolate either change. It also didn't address names, sizes, years or abstention. Round 1 had no preserved workshop Clairic run, so we generated a fresh naive baseline for the text comparison. Treat all the figures as directional, not as a model leaderboard.
Try it on your own data
The best test of either approach is your own documents. The harness we built for the workshop runs both models against the same schema and the same data, then scores them field by field. If you'd like to see how a purpose-built extraction model compares with an LLM on your own documents, get in touch and we'll run it with you.
Clairic details are from Cloudflare's announcement and model reference. Museum images are from The Met's Open Access collection (CC0). Monster text is from the D&D SRD 5.1 (CC-BY 4.0).
