I measured my framework on features we had already shipped: 4 features from S to XL, fragile line references 64 to 0, plans 21% shorter, 413 tests green.

I measured my framework on features we had already shipped

In June I changed how claude-agentic-kit, the framework Trusq is built with, hands a specification over to a plan. Before trusting the change, I wanted evidence from real work, not from the kit's own examples. So I took four Trusq features that were already in production, regenerated their plans with the new version and compared the results with what we had actually built. Fragile line references in the plans fell from 64 to 0, the plans became 21% shorter, and one feature built end to end from a new plan avoided two issues that a human reviewer had caught on the real pull request.

What I changed

In the kit, every feature goes through the same steps: refine, plan, implement, review, document. The refined document is the specification, and the plan says which code changes in which pull request. The change I wanted to test sat between these two steps, and it had two parts.

The first was a ban on line numbers. Plans written by agents liked to point at code as sender.go:115-121. That is precise on the day the plan is written and wrong a few commits later, and an agent that follows such a reference edits the wrong lines without noticing. A gate script now rejects every line reference before the first review round, and the plan has to name a function or a type instead.

The second was a rule: reference, never restate. Plans used to copy whole passages of the specification, and every copy is a place where the two documents can drift apart. Now the plan points to the specification instead of repeating it.

Why shipped features make a fair test

The Trusq repository still carried the kit from before the change. So its delivered plans were a ready-made baseline from the old process, and the regenerated plans were the candidate. The model was the same on both sides, Claude Opus, so any difference came from the process and not from the model. And the code we had shipped, along with the human review of it, was ground truth that neither version had seen when it wrote its plan.

I picked four features, one from each size class the kit uses: a small one that closes a conversation when the customer says they are satisfied, a medium one that checks every citation in an AI reply, a large one that unified delivery status across five channels, and an extra-large one that set up the admin backend with its login.

What came out

Delivered plansRegenerated plans
Fragile line references640
Length in lines1,5711,244 (−21%)
Matches with the shipped code47
Divergences from the shipped code17

The line references were a real hazard, not a style issue: at least two of those in the large feature's plan already pointed at the wrong lines, because the code had moved since the merge. Across all 14 plans delivered in the Trusq repository at that point, there were 114 of them.

The small, medium and large plans shrank by 42%, 31% and 21%, purely because they stopped copying the specification. The extra-large one grew by 29%, because its delivered plan had been unusually short for a feature of that size, and the new one was more complete. So the gain is that nothing is restated, not fewer words at any price.

Every one of the 17 divergences was a choice made during implementation that neither plan had predicted, such as a filter for personal data that was added while building the large feature. On the small feature the new plan was even more accurate than the delivered one: it left open whether a status value already existed, where the old plan had wrongly claimed it was missing.

One feature, end to end

I took the small feature all the way through. From the regenerated plan and the specification alone, without looking at the shipped code, the agents wrote the code test-first: 637 lines in six files. The build passed and all 413 tests ran green, 12 of them new. The adversarial review found one real issue in its first round and declared the code clean after three.

Then I compared the result with the real pull request, which a colleague had reviewed. Two of the colleague's findings had never entered the generated code. The real code asked the model for a resolution status even on a customer's first message, where it means nothing; the generated code only asks on follow-ups, and a test checks that. And the real code lacked a test proving that a first reply never closes a conversation; the generated code had it from the first commit. The new plan had already stated the follow-up rule as a design decision, so the mistake had no way in.

Checking the measurement

Three times during this run, a tool reported a green result that was wrong. The line-reference check first reported 0 for every delivered plan, because it refused files outside its allowed path and the count only picked up the error message. Each time, measuring again by a second route caught it, and the report records all three. I trust a measurement of an agent's work only once something other than the agent has repeated it.

What this does and does not show

Four features and one end-to-end run are enough to show that the effect is real across every size class. They are not a statistical proof, and the report says so in the same words. The next step is to build the new structure into the planning template and repeat the comparison on a larger sample.

If you change how your team or your agents work, test the change on work whose outcome you already know. Shipped features come with their own answer key: the code that was built and the review it got.