ParseBench-100

ParseBench-100 is a test that measures how well AI reads text in scanned documents or images. Just as a person reads a document with their eyes, it gives AI a document and scores whether it can correctly extract the text and tables inside.

Even with the same test, scores can vary widely depending on the execution framework surrounding the AI. Design choices — like whether the document is split into small chunks before reading, or whether tables and images are processed separately — show up directly in the score. This test has also been used to check whether documents are processed entirely on-device without leaving the machine.

The article reports that "on ParseBench-100, which tests document reading, the portable computer scored 65.1%, far ahead of Hermes (34.6%) and Pi (13.9%), with the entire process handled on-device so documents never leave the machine." A common misunderstanding here is assuming the score gap reflects differences in the AI models themselves — but in reality, even the same model can score very differently depending on how the execution framework around it is designed.