The Packkit bugs I only found by using it

Packkit has a large test suite, and for a while it gave me a false sense of safety. The generator would pass everything, then I would scaffold an actual project to use, push it, and watch its CI fail on the first run for a reason none of my tests looked for. The tests checked that the config was computed correctly. They did not check that the correct config produced correct files. That gap is where every real bug lived.

This post is about four of those bugs and the one discipline that came out of them.

Four bugs, one shape

The bugs were different on the surface and identical underneath.

The full stack preset accepted --lint biome, stored it on the config, and then emitted ESLint anyway. The monorepo code path was written before the lint option existed and never read it. So the setting was persisted, correct, and ignored. You asked for Biome and got ESLint with no error.

Single-package projects using pnpm shipped without a packageManager field in package.json. The generated CI used pnpm/action-setup, which reads the pnpm version from exactly that field, so the first CI run died with "No pnpm version is specified". The monorepo presets set the field; the single-package ones did not, and nothing noticed.

Yarn projects were worse. The generated CI ran yarn install --immutable, which is modern Yarn syntax, but nothing provisioned modern Yarn, so on a runner with the classic Yarn 1.x preinstalled the command was not even recognized. The scaffold looked like it supported Yarn and did not.

And the React app preset had six problems at once, found in a single afternoon of using it: no Vite client types so tsc failed on import.meta.env, a Biome config that scanned the build output and reported thousands of errors after a build, a main entry that tripped its own linter, a schema URL pinned behind the installed tool, a summary line that named the wrong build tool, and a provenance baseline with its dependency sections nested one level too deep.

Every one of these passed the existing tests, because the existing tests asked "is the config right" and the config was always right. The generation was wrong.

The discipline: assert on the files, and make the test bite

The fix was not just the fixes. It was a new kind of test that asserts on the generated output, not the config, and sweeps the option combinations where these bugs live: preset by linter by package manager. It checks the things a config test cannot see. The chosen linter's config is present and the others' are absent. The installed lint tooling matches the choice. No script invokes a tool that is not a dependency. A pnpm project has a packageManager field. A Yarn project is set up for modern Yarn. CI runs a lint step exactly when there is a lint script to run.

There is one rule I hold these to: a regression test has to be proven to fail on the bug. After I write the fix and the test, I put the bug back, run only that test, and confirm it goes red. Then I restore the fix. A test that stays green whether or not the bug is present is not protecting anything, it is decoration, and the only way to know the difference is to make it fail on purpose once.

That check paid off immediately. When I re-introduced the lint bug, three of the invariants went red. When I dropped the pnpm field, its guard went red. Each fix left behind a test I had watched catch the exact thing it was there to catch.

The count today is twelve of these output invariants, and the full suite is 164 tests. The invariants run in about a quarter of a second, because they generate in memory and read the file map, no installs, so sweeping every preset and linter and package manager is cheap enough to keep doing.

What the fast tests still cannot tell me

Here is the honest edge. These invariants prove the generator writes the right files. They do not prove those files actually install and run. They confirm a Biome project has a Biome config that excludes the build output; they do not run Biome. They confirm a Yarn project pins modern Yarn; they do not run a Yarn install.

That last mile is a different kind of test, slower and network-bound: scaffold the project, install its dependencies, run its linter, and see it exit clean. I have one of those planned, deliberately scoped to a single case, the full stack Biome preset, because the value is in catching "the files are right but they do not run together" and one honest case catches that class. I have resisted turning it into a matrix, because the fast tests already cover generation correctness and every added install costs real minutes.

The larger lesson is the boring one. I found these by using the thing, not by testing it, and no amount of testing the config would have surfaced them. So now the rule is that using it counts as testing, and anything I trip over in real use becomes an invariant that would have caught it, proven by watching it fail. Four bugs, four tests that bite, and a generator I trust a little more because I stopped trusting it to be correct on my say-so.

The generators and the test suite are on the PackkitLabs org, and the configurator that runs all of this is at packkit-web.pages.dev.