Long documents still lose the plot
Ai2's StateBench checks whether a model can track changing facts across a long document, at four levels of difficulty. Frontier models handle the easy levels and break down on the hardest long ones. It is a benchmark. It will not file your contracts.
- Verdict
- Watch Promising, but something is missing: access, pricing, or proof.
- For
- people judging long-document models
- Why it matters
- Frontier models handle the easy levels and break down on the hardest long ones. It will not file your contracts.
- Next check
- Recheck if a product you use cites a StateBench result for the documents you care about.
- Published
- Filed
- By
- The Loaded Tokens desk
- Topic
- Search & research
- Sources
- Allen Institute for AI
Sources & limits: This is our recap, not a copy of the original. The links above are the pages we read. We didn't test it ourselves. How we verify
Corrections: if a fact is wrong, email hello@loadedtokens.com.