Retry the Harbor manifest copy instead of a probe for it #3

Merged
magnus merged 1 commits from fix-harbor-tag-retry into main 2026-10-11 07:58:51 +00:00
Owner

Fixes the red build on main from the last merge. CI-only; no application code changes.

What failed

The image was pushed and deployed fine. The failure was the final step, adding the
companion main-<timestamp>-<sha> audit tag:

412 Precondition Failed
current image with "Running" status of vulnerability scanning cannot be pulled
due to configured policy in 'Prevent images with vulnerability severity of "High" or higher from running.'

Two faults compounded

  1. The loop gated on the wrong command. It probed with imagetools inspect, but the
    operation that needs permission is imagetools create, which copies the manifest.
    Harbor let the probe through while still blocking the copy, so the gate opened early.
  2. create sat outside the if condition. Under the default bash -e its first
    failure aborted the step before reaching exit 0 or another attempt. The log proves
    it: exactly one waiting for Harbor scan/pull policy line, then the error — 58 of 60
    retries unused.

The fix

Retry the real copy, inside the condition, so a blocked copy is a reason to wait rather
than fatal.

Verified by simulating both loops under bash -e with a stub that mirrors the real
sequence — the probe clearing on attempt 2 while the copy stays blocked until attempt 3:

OLD: waiting (1/5) -> 412 -> step exit 1   (companion tag never created)
NEW: 412 -> waiting -> 412 -> waiting -> created tag -> step exit 0

The old path reproduces the production log exactly. YAML parses, bash -n is clean, and
make test passes (12). The real proof is the next merge to main.

Trade-off

A permanently failing copy now takes the full ten minutes to report instead of failing
fast. Worth it: the previous behaviour failed fast but wrongly, on a transient block. I
avoided matching Harbor's error text to tell the two apart, since that couples CI to an
error string.

Also

CLAUDE.md/AGENTS.md now record the two-tag publish and, more usefully, that only the
primary tag matters for rollout — the Flux image policy matches ^main-<14 digits>$ — so a
failure in this step costs the audit tag, not availability. That was not obvious while
reading a red build against a healthy production.

🤖 Generated with Claude Code

Fixes the red build on `main` from the last merge. CI-only; no application code changes. ## What failed The image was pushed and deployed fine. The failure was the final step, adding the companion `main-<timestamp>-<sha>` audit tag: ``` 412 Precondition Failed current image with "Running" status of vulnerability scanning cannot be pulled due to configured policy in 'Prevent images with vulnerability severity of "High" or higher from running.' ``` ## Two faults compounded 1. **The loop gated on the wrong command.** It probed with `imagetools inspect`, but the operation that needs permission is `imagetools create`, which *copies* the manifest. Harbor let the probe through while still blocking the copy, so the gate opened early. 2. **`create` sat outside the `if` condition.** Under the default `bash -e` its first failure aborted the step before reaching `exit 0` or another attempt. The log proves it: exactly one `waiting for Harbor scan/pull policy` line, then the error — 58 of 60 retries unused. ## The fix Retry the real copy, inside the condition, so a blocked copy is a reason to wait rather than fatal. Verified by simulating both loops under `bash -e` with a stub that mirrors the real sequence — the probe clearing on attempt 2 while the copy stays blocked until attempt 3: ``` OLD: waiting (1/5) -> 412 -> step exit 1 (companion tag never created) NEW: 412 -> waiting -> 412 -> waiting -> created tag -> step exit 0 ``` The old path reproduces the production log exactly. YAML parses, `bash -n` is clean, and `make test` passes (12). The real proof is the next merge to `main`. ## Trade-off A *permanently* failing copy now takes the full ten minutes to report instead of failing fast. Worth it: the previous behaviour failed fast but wrongly, on a transient block. I avoided matching Harbor's error text to tell the two apart, since that couples CI to an error string. ## Also `CLAUDE.md`/`AGENTS.md` now record the two-tag publish and, more usefully, that only the primary tag matters for rollout — the Flux image policy matches `^main-<14 digits>$` — so a failure in this step costs the audit tag, not availability. That was not obvious while reading a red build against a healthy production. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
magnus added 1 commit 2026-10-11 07:56:45 +00:00
Retry the Harbor manifest copy instead of a probe for it
pull-request / validate (pull_request) Successful in 32s
7ea25ee4fd
The companion SHA tag step failed on the last build even though the image had
already been pushed and deployed. Harbor's vulnerability policy blocks manifest
reads while Trivy's scan is Running, and `imagetools create` copies the manifest,
so Harbor answered 412 Precondition Failed.

Two faults compounded. The loop gated on `imagetools inspect`, but a probe passing
is no guarantee the copy will be allowed. And `create` sat outside the condition,
so under the default `bash -e` its first failure aborted the step before `exit 0`
or any further attempt: the log shows one waiting line, then the error, with 58 of
60 retries unused.

Retry the real operation instead, inside the condition so a blocked copy is a
reason to wait rather than fatal. The trade-off is that a permanently failing copy
now takes the full ten minutes to report instead of failing fast; the previous
behaviour failed fast but wrongly, on a transient block.

Also records that only the primary tag matters for rollout, since the Flux image
policy matches the timestamp tag, so a failure in this step costs the audit tag
rather than the deployment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
magnus merged commit 803df14813 into main 2026-10-11 07:58:51 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: magnus/btcprice#3