tokensift
An open-source linter that scans your LLM prompts for token waste — like UUIDs, pretty-printed JSON, and repeated text — and estimates the real dollar cost of each finding.
🔗 Visit tokensift
Description
If you've ever pasted a wall of pretty-printed JSON or a long UUID into a prompt without thinking twice, tokensift is built to catch exactly that habit. Since AI APIs charge by the "token" — small chunks of text — and token counts don't line up neatly with how many characters something has, small habits like extra whitespace or verbose IDs quietly inflate your bill every single time that prompt runs. tokensift works like a code linter, in the spirit of ESLint, except instead of flagging bad code style, it flags bad prompt style — pointing out, for instance, that a UUID costs 18 tokens when a short ID would cost 3.
Technically, tokensift is a zero-dependency TypeScript library and CLI that tokenizes prompts offline and deterministically, using OpenAI's actual BPE vocabularies for exact counts on GPT-4o/4/3.5/o1/o3, and calibrated estimates (about 7.6% mean error) for Anthropic's Claude models. It ships 20 built-in detection rules — for repeated blocks, base64 blobs, and filler text among others — and translates each finding into an actual dollar-cost estimate, not just a token count. It supports custom rules, CI integration with GitHub/SARIF/Markdown output, baseline regression tracking, per-file token budgets, and test matchers for vitest and jest. It runs anywhere JavaScript runs: Node.js, Deno, and edge runtimes like Cloudflare Workers.
💬 Our review
The short version: a genuinely useful, no-nonsense idea — an ESLint-style linter for wasted prompt tokens — executed cleanly, though it's still a small, young project you should pilot before betting a CI pipeline on it.
tokensift doesn't really compete with prompt-management platforms like PromptLayer or Langfuse, which focus on logging, versioning, and observability across a whole LLM application — it does one narrower job well: catching token waste before a prompt ships, the way a linter catches bugs before code ships. That's a real gap; most teams either eyeball their prompts or rely on generic tokenizer counters that tell you how many tokens without telling you why so many or what it's costing you. The exact OpenAI tokenization is a genuine strength; the Claude numbers being estimates, not exact, is a fair caveat the docs are upfront about. At 8 stars and MIT-licensed with zero dependencies, there's no cost barrier to trying it — the real question is whether its 20 built-in rules cover the specific waste patterns in your own prompts, which you'll only know by running it.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Free and open source under the MIT license; published as an npm package, no paid tier
Pros
Free, open source (MIT), zero runtime dependencies
Exact token counts for OpenAI models via real BPE vocabularies
Translates findings into actual dollar-cost estimates, not just token counts
CI-ready: SARIF/GitHub/Markdown output, vitest/jest test matchers, baseline tracking
Cons
Very young project — only 8 GitHub stars, limited real-world track record
Claude/Anthropic token counts are calibrated estimates, not exact (~7.6% error)
Only 20 built-in rules — may not cover every waste pattern out of the box
No hosted dashboard or team features — it's a CLI/library, not a SaaS platform