What we measured, and what it taught us
The guides tell you what Krypton does. This page keeps the experiments behind them: one open-source project, garth, tested in depth, and 19 more repositories for the prompt-caching survey.
Prompt caching on the live Claude API
We replayed 12 real commits of garth (about 40,000 tokens of source per turn as Claude counts them) against the live Claude API (Sonnet 5.5, four cache breakpoints per request) and measured how much of each prompt the provider reused and what the replay cost.
| File order | Predicted reuse | Measured reuse | Cost of the 13 turns |
|---|---|---|---|
| Krypton (most stable first) | 73% | 71% | $0.51 |
| Alphabetical | 48% | 47% | $0.79 |
| Reverse (most volatile first) | 0% | 0% | $1.30 |
Excluding the first turn, which can never hit the cache. Including it, Krypton's order reused 65.7%. Krypton's order made the replay about 35% cheaper than alphabetical order. With a single cache breakpoint at the end of the prompt, every order reused 0%, so the ordering only pays when the stable files sit in front of earlier breakpoints. Predictions come from the shared prefix of consecutive prompts and landed within about two points of the live numbers.
Does it hold on other repositories?
Not always. We ran the same replay on a second repository, pluggy, against the live Claude API, and Krypton's order did worst:
| File order | Predicted reuse | Measured reuse |
|---|---|---|
| Krypton (most stable first) | 4% | 4% |
| Alphabetical | 23% | 22% |
| Reverse (most volatile first) | 26% | 25% |
The live measurements on garth and pluggy were within one point of the predictions. The table compares predicted reuse across 20 repositories, ranked by Krypton's gain over alphabetical order. These results show why Krypton evaluates several candidate orders against repository history: it chooses the strongest candidate only when the replay predicts an improvement over your current order. If no candidate clears that bar, the request keeps its original order.
| Repository | Files | Krypton | Alphabetical | Reverse | Krypton vs alphabetical |
|---|---|---|---|---|---|
| pallets/flask | 24 | 51% | 20% | 18% | +31 |
| pallets/click | 17 | 27% | 0% | 32% | +27 |
| matin/garth | 41 | 73% | 48% | 0% | +25 |
| pallets/itsdangerous | 8 | 45% | 29% | 17% | +16 |
| pallets/jinja | 25 | 19% | 6% | 17% | +13 |
| psf/requests | 19 | 23% | 12% | 31% | +11 |
| encode/httpx | 23 | 27% | 16% | 33% | +11 |
| Textualize/rich | 100 | 49% | 38% | 21% | +11 |
| pallets-eco/flask-wtf | 10 | 42% | 32% | 14% | +10 |
| hynek/structlog | 21 | 19% | 14% | 28% | +5 |
| pypa/packaging | 22 | 35% | 33% | 37% | +2 |
| tqdm/tqdm | 31 | 22% | 20% | 40% | +2 |
| pallets/markupsafe | 2 | 33% | 33% | 33% | +0 |
| agronholm/anyio | 46 | 40% | 41% | 32% | −1 |
| theskumar/python-dotenv | 8 | 33% | 36% | 27% | −3 |
| python-attrs/attrs | 19 | 11% | 14% | 43% | −3 |
| python-jsonschema/jsonschema | 37 | 14% | 17% | 37% | −3 |
| python-attrs/cattrs | 50 | 25% | 37% | 38% | −12 |
| pallets/werkzeug | 54 | 23% | 36% | 11% | −13 |
| pytest-dev/pluggy | 8 | 4% | 23% | 18% | −19 |
- Krypton's order beat alphabetical on 12 of 20, tied on 1 and lost on 7.
- It was the best of the three orders on only 7.
- On average it reused 31% of each prompt against 25% for alphabetical and 26% for reverse. The typical (median) gain over alphabetical was 4 points, and garth is one of the best cases.
krypton order . --subdir src --model gpt-4o --post. It replays recent commits against the candidate orders without running a model or sending source.Does the ordering guard pick well?
We tested the decision on garth without peeking: the decision is made from history alone, then the next 12 commits are replayed with exact token prefixes. This was repeated from 14 starting points, and for three kinds of caller order.
| Your order | Reordered | Gain when reordered | Reordered and worse | Kept but would have gained |
|---|---|---|---|---|
| Alphabetical | 13 of 14 | +13.4 points | 1 | 1 |
| Shuffled | 10 of 14 | +27.1 points | 0 | 4 |
| Worst case (most volatile first) | 14 of 14 | +47.9 points | 0 | 0 |
Two things to read from it. When your order is already poor, it reorders and the gain is large. When it hesitates, it keeps, which in the shuffled case means missing some gains: the guard is deliberately conservative, because a missed gain costs nothing and a bad reorder does.
Refactoring and the payback
A Krypton-guided refactor of garth took the logic-complexity flags from 9 to 0 with all 163 tests passing, where an unguided refactor left 8 of 9. We then asked whether the cleaned code is cheaper for an agent to change. Fresh agents made the same five change tickets on the original code and after each of two cleanups, six runs each, with real token usage priced at list prices:
- Cleaning everything cost $0.48 once and made each later change 16.9% cheaper on average, but the typical run saved about 6% (runs: 44%, 7%, 5%, 25%, 0%, 1%).
- Cleaning only the often-changed, tangled files (chosen from git history) cost $0.27 and saved 8.0% on average (runs: 26%, 14%, −17%, 20%, 0%, −12%).
- On the averages, cleaning everything pays for itself after about 66 changes and the hot-files option after about 79.
- Without a cleanup, one feature kept compounding: after five revisions its main function was about 457 times over Krypton's limit, from 46 times at the start. With a cleanup first it stayed between 5 and 37 times over.
- Rebuilding garth worst-function-first until Krypton scored it 81 (from 62) left the cost of the same five later revisions unchanged in one run ($0.176 against $0.177).
| Changes after the cleanup | Net dollars saved | Return on total spend |
|---|---|---|
| 10 | −$0.40 | −94.0% |
| 25 | −$0.29 | −27.5% |
| 66 (break-even) | +$0.00 | +0.1% |
| 100 | +$0.25 | +5.8% |
| 250 | +$1.34 | +12.5% |
| 500 | +$3.15 | +14.7% |
Return on total spend includes the one-time cleanup, so it approaches 16.9% as changes accumulate. The table uses the clean-everything averages.
A CI gate over twenty revisions
We also tested Krypton as a check in CI. One feature of garth grew over 20 sequential revisions made by an agent, three times with each setup. In the plain setup the agent just did each ticket. In the gate setup the harness scored the code with Krypton after each ticket, and if a function the ticket touched was over the structure limit, an agent was sent back with Krypton's report to fix it before the next ticket. Neither agent was given the Krypton tool or told to refactor.
| Plain | With the gate | |
|---|---|---|
| Cost of 20 revisions | $0.92 | $1.49 (+62%) |
| Worst function at the end | about 1,900× over the limit | about 12× over |
| Structure score at the end | 55 | 72 |
| Revisions done correctly | 60 of 60 | 60 of 60 |