Assets & Wealth Extraction Benchmark
The AWE (Asset and Wealth Extraction) Benchmark scores how accurately models pull names, dates and numbers out of the documents the wealth & equity industry runs on. Every PDF is synthesised with an exact answer key, so nothing is scraped or memorised from training data. Each field is scored as correct, missing, or one of several kinds of wrong. Accuracy is correct fields over scored fields; a document that fails outright counts all its fields as missing.
Overall accuracy
Correct fields as a share of scored fields, across all sections and field kinds. The chips are the reasoning level (the effort the run was sent, or, in muted type, the vendor's documented default when none was sent) and the input the model was given; hover either for detail. Outlined bars are models where at least one document failed and dragged the score down.
Cost per document
USD per document at the list prices recorded in the run config. Local models run at no marginal cost and are not shown.
Cost against accuracy
USD per document at the list prices recorded in the run config, on a log scale. Local models run at no marginal cost and are left out. Hover a point for the model.
Median seconds per document on a log scale. Hover a point for the model.
Accuracy by field kind
Names use fuzzy matching (legal suffixes ignored); dates must match to the day, or to the month where the letter prints only a month; numbers must be within a small tolerance in the document's own units.
Accuracy by document family
Families follow the kinds of document a fund sends: investor reports, capital statements, financial statements, subscriptions and redemptions, capital calls and distributions.
Accuracy by document layout
Each archetype is a distinct letter style: bullet lists, newsletter cards, formal financial statements, a long-form report with an appendix, and a short community brief. Hover a cell for counts.
Accuracy by section
Where in the letter the field lives. Financial statements and portfolio cards are table-heavy; highlights and events are prose.
Why fields are missed
Wrong answers as a share of scored fields, split by failure kind. Missing is a null or an unpaired list item; scale is off by a power of ten (a thousands table read literally); rounding is within 5% but outside tolerance; near date is within three days.
Hallucinations and invented items
Values given for fields the letter does not state, and list entries (investments, exits, rows) that do not exist. Neither counts against accuracy, so they are shown separately.
All models
The table view of everything above. Context is the window the model was served with; effort is the reasoning level and the vendor parameter it was sent as, or the vendor default that applied when none was sent.