Summary
- Perplexity said the 27B on-device model behind its local agent "Portable Computer," combined with a purpose-built harness, beat open-source harnesses Pi and Hermes on knowledge-work benchmarks
- On web search (BrowseComp) and document parsing (ParseBench-100), it scored higher while using less time and fewer tokens, with everything but search running entirely on-device
- The company said routing to a frontier model for text-only advice — with user approval, only when needed — pushed Terminal Bench 2.1 scores from 59.6% to 73.0%
A local agent that beat open-source harnesses
On August 25, Perplexity followed up its earlier announcement — covered in Perplexity Launches a Cloud-Free Agent on DGX Spark — of a fully local agent called Portable Computer with a separate research thread detailing benchmark numbers. The core finding: pairing a 27B-class on-device model with a purpose-built harness — the execution framework that governs when and how a model uses tools — outperformed open-source competitor harnesses Pi and Nous Research's Hermes on real knowledge-work tasks. Perplexity explained that "small models fail on harnesses built for frontier models," and said it redesigned the harness itself to match the model's scale.
Same model, different harness, different scores
A notable result came from the "Local Knowledge Work Bench" test, which ran 53 tasks three times each for 159 total rollouts. All three harnesses used the same Qwen3.8 27B model, yet scores diverged sharply.
| Harness + Model | Score |
|---|---|
| Portable Computer + PPLX 27B (post-trained) | 85.4% |
| Portable Computer + Qwen3.8 27B | 82.6% |
| Pi + Qwen3.8 27B | 77.6% |
| Hermes + Qwen3.8 27B | 74.0% |
According to Perplexity, feeding in the same model produced up to an 8.6 percentage-point swing depending on harness design alone — and adding PPLX 27B, a version post-trained for this specific purpose, pushed the score up to 85.4%.
Search stays outside, everything else stays inside
On BrowseComp, a 1,266-task benchmark for web search ability, Portable Computer scored 66.7%, ahead of Pi (50.2%) and Hermes (43.9%), while also finishing faster — around 400 seconds on average versus roughly 800 for Pi and 1,000 for Hermes. Token usage was lower too, at 852k compared to Pi's 2.82M and Hermes's 1.01M. On ParseBench-100, a document-reading benchmark, Portable Computer scored 65.1%, well ahead of Hermes (34.6%) and Pi (13.9%) — and the company says this entire process runs on-device, so documents never leave the machine. Reasoning and personal documents stay local; only search reaches out to the web.
| Benchmark | Portable Computer | Pi | Hermes |
|---|---|---|---|
| BrowseComp (1,266 tasks) | 66.7% | 50.2% | 43.9% |
| ParseBench-100 (document OCR) | 65.1% | 13.9% | 34.6% |
When stuck, it asks a frontier model — in text only
When the local model can't solve a problem on its own, it can ask a frontier model for advice, but only with user approval. If personal information is present, it's flagged, and the frontier model never gets direct access to files or tools — it only returns text guidance. Turning on this escalation path raised Terminal Bench 2.1 scores from 59.6% to 73.0%, at a cost of $0.415 per rollout. Perplexity framed this as closing roughly three-fifths of the gap to frontier models at about two-thirds of the cost.
Why the model and harness were built together
Portable's execution framework is built around a minimal system prompt, skills that load only when needed, lightweight CLI-based connectors in place of MCP servers, self-verification where the model checks its own outputs, and an always-on sandbox. PPLX 27B was post-trained inside this harness using synthetic tasks modeled on real usage patterns — Perplexity said no actual user data was used. The company said that as models, chips, and devices keep improving, local-first performance will keep improving alongside them, and it plans to release the benchmarks used here soon.
Portable Computer is the fully local version running on NVIDIA's DGX Spark, available to Perplexity Pro and Max subscribers, as covered in the earlier article. This research fills in the detailed numbers behind those performance claims.
Editor's view
Perplexity deliberately ran the same model — Qwen3.8 27B — through three different harnesses to make one point clearly: harness design, not model size, is what drives performance. The comparison also directly targets open-source players Pi and Nous Research's Hermes, which reads as a signal that competition among local agents is shifting from a race over model size to a race over execution-framework design. Set against recent reports that NVIDIA is negotiating an investment in Perplexity at a valuation north of $30 billion, it's easier to see why Perplexity built a harness optimized specifically for DGX Spark hardware.
Anyone who's tried putting a 27B-class model to real work knows the usual wall: it's not as sharp as a cloud frontier model, and it starts to stumble the moment tool calls get even slightly complex. What Portable Computer demonstrates is an alternative to scaling the model up — resizing the harness to fit the model instead — and adding text-only escalation to a frontier model, used only when needed, closes much of that gap at a fraction of the cost. For legal or medical teams wary of uploading entire documents to the cloud, this kind of setup could offer a real alternative to cloud-based agents.
That said, all these numbers come from one specific piece of hardware, DGX Spark — whether the same performance holds on an ordinary laptop or phone is a separate question. Since Perplexity has promised to release the benchmarks, expect outside developers to start reproducing and stress-testing these numbers within the next few weeks.
Comments