Keep large tool output out of LLM context: 3x accuracy 95% fewer tokens

Details

External ID

47261565

Source

HN

Company

—

Product

Keep large tool output out of LLM context: 3x accuracy 95% fewer tokens

Website domain

github.com

Launched

March 5, 2026

Cohort

—

Upvotes

10

Upvotes percentile

0.5781057810578106

Tags

—

Fetched at

Sept. 7, 2026, 9:26 p.m.

Updated at

Sept. 7, 2026, 9:26 p.m.

Description

LLM agents often place raw JSON tool outputs directly in the prompt. After a few tool calls, earlier results get compacted or truncated and answers become incorrect or inconsistent.I built Sift, a drop-in MCP gateway that stores tool outputs as local artifacts (filesystem blobs indexed in SQLite) and returns an `artifact_id` plus compact schema hints when responses are large or paginated.Instead of reasoning over full JSON in the prompt, the model runs a small Python query: def run(data, schema, params): return max(data, key=lambda x: x["magnitude"])["place"] Query code runs in a constrained subprocess (AST/import guards + timeout/memory caps). Only the computed result is returned to the model.Benchmark (Claude Sonnet 4.6, 103 questions across 12 datasets):- Baseline (raw JSON in prompt): 34/103 (33%), 10.7M input tokens- Sift (artifact + code query): 102/103 (99%), 489K input tokensOpen benchmark + MIT code: https://github.com/lourencomaciel/sift-gatewayInstall: pipx install sift-gateway sift-gateway init --from claude Works with Claude Code, Cursor, Windsurf, Zed, and VS Code. Existing MCP servers and tools require no changes.

• Source listing
• Website