Trace a test before you trust its cache hit

A Go test that prints (cached) reused an earlier result under the inputs the Go command tracked. Go documents those cache conditions. That saves time. It also leaves a question when the test starts another program, discovers files at runtime, or consults a service: would a change in that input make the test run again?

We use a local CI/lock trace to investigate one suspicious test at a time. Source inspection supplies the candidate. The trace records what its backend observed during a real execution. An input-change experiment decides whether the cache rule is safe.

Capture one bounded test

Check the installed binary's cilock run --help-advanced first. From the Go module directory, use a private temporary directory and a throwaway local key. The run below stays offline and executes the selected test once on purpose:

trace_dir=$(mktemp -d /tmp/cilock-test-trace.XXXXXX)
openssl ecparam -name prime256v1 -genkey -noout -out "$trace_dir/local.key"
cilock run --offline --enable-archivista=false \
  -k "$trace_dir/local.key" --trace --capture-mode trace:auto \
  --workload manual --attestations command-run \
  --step test-side-effects --workingdir "$PWD" \
  --outfile "$trace_dir/tests.dsse.json" \
  -- go test ./internal/package -run TestSuspect -count=1 -timeout=60s

command-run is built in; naming it explicitly keeps automatic project attestors out, though this CI/lock build warns that it is already built in. The envelope can contain command arguments, paths, and test output. Keep it local and inspect only the fields you need.

Read the observation before drawing a conclusion

Decode the DSSE payload to inspect the command attestation's summary and process count. Decoding is not signature verification:

jq '.payload | @base64d | fromjson | .predicate.attestations[] |
  select(.type | contains("/command-run/")) |
  .attestation | {summary, exitcode, processes: (.processes // [] | length)}' \
  "$trace_dir/tests.dsse.json"

Read the reported backend and diagnostics first. Then inspect the process and network records in the local envelope. Repeated Git, Node, or compiler launches can explain setup cost. A connection is a lead to investigate with the test source and transport; a local HTTP fixture may have produced it. File observations can reveal an input or write outside the test's private scratch directory when the backend supports that observation.

Challenge the cache key

A real case in our Go suite involved an Ent schema inventory test. Its loader discovered schema files, but adding an otherwise unclassified schema file left the existing test at (cached). An inventory-only import made that schema package an explicit test dependency. With the same added file, the test reran and failed as intended. After removing the file, it passed; the next unchanged run was cached. Source review and that new-file experiment established the cache escape; the focused fix is in PR #9689. A CI/lock trace did not discover it.

For a suspect test, remove -count=1 and run the exact package command twice. Require an actual (cached) result. Change one relevant input, run it again, restore the input, and repeat. Use a focused GODEBUG=gocachetest=1 run to explain a miss. Compare traced runs with traced runs and ordinary runs with ordinary runs when measuring time; tracing has its own cost.

If an external service or subprocess state remains outside the cache key, keep that test fresh or supply a deterministic fixture before sharing results across worktrees. Preserve the test's regression assertion while changing its setup.

Keep the evidence roles separate

The throwaway key above signs a local diagnostic artifact. It is not the commit-bound test evidence a Pushgate policy may require. Pushgate evaluates the exact signed evidence and policy for a push; a useful trace does not turn an unsafe cache hit into a passing release.