btcprice/btc-price-deploy.md

# btcprice GitOps migration plan

**Target path**

```text
git push (magnus/btcprice)
  -> Gitea Actions on anvil
  -> Harbor image + Trivy scan
  -> Flux image policy commits magnus/deploy
  -> Flux reconciles k3s
  -> ingress-nginx
  -> hostinger Traefik TLS
  -> https://btcprice.k8s.pmagnus.cloud
```

Production deployment must not use `docker compose`, `kubectl apply` from CI, or a CI-held
kubeconfig. The existing Compose instance on citadel is retained only during validation and as a
short rollback path.

## Access and secret handling

Use `/Users/pmagnus/pi/system/.env` for Gitea, Harbor, host and Cloudflare credentials. The file is
mode `0600` and must remain outside Git.

```bash
set -a
. ../system/.env
set +a
```

Rules:

- never echo tokens/passwords or put them in command arguments;
- pass secrets via API headers, stdin, or temporary `0600` files;
- use the Harbor CI robot for image pushes, the existing cluster pull robot for pulls, and Harbor
  admin only for project policy changes;
- use the Gitea token for API administration, not from application or workflow code;
- do not copy `../system/.env` into this repository, an image, a pod, or `magnus/deploy`.

## Audited starting point (2026-09-15)

Already working:

- `magnus/btcprice` uses `main`; `.gitea/workflows/build.yml` tests and publishes immutable amd64
  images from `anvil`.
- The latest Gitea run for commit `dd67eb7` succeeded.
- Harbor project `btcprice` is private, scans on push, blocks High-or-worse vulnerable images, has a
  retention policy and weekly garbage collection. The latest image is scanned successfully:
  `main-20260915160353` / `main-20260915160353-dd67eb7`.
- The repository has `HARBOR_ROBOT_USER` and `HARBOR_ROBOT_TOKEN` Actions secrets.
- The image exposes `/api/health`, dependency-aware `/api/ready`, and `/api/version`.
- Flux and all image automation controllers are healthy. The `hello` canary proves unattended image
  update, reconciliation, two-node scheduling, ingress, and TLS.
- k3s has two Ready amd64 nodes: `srv1893094` and `citadel`.
- Both nodes already have Harbor pull credentials.
- ingress-nginx is managed by Flux and is reached through hostinger Traefik on NodePort 30080.
- btcdata still runs under Compose on citadel and answers on `100.75.8.131:4102`.
- The old btcprice Compose container remains available on citadel port 4041.

Current gaps:

- `magnus/deploy` has no btcprice resources or image policy.
- `btcprice.k8s.pmagnus.cloud` currently reaches Traefik but receives its default certificate and an
  nginx 404; the TLS domain and Kubernetes Ingress do not exist yet.
- Local `../deploy` can lag Flux's automated commits. Always run `git pull --ff-only` before editing.
- Harbor's release retention rule currently keeps `v*`, while the workflow strips the leading `v`
  (`v1.2.3` becomes `1.2.3`). Correct that mismatch before relying on release retention.
- The Gitea btcprice repository is currently public. Confirm that this is intentional; it does not
  expose the Actions secrets, but it does expose source and workflow definitions.

## Phase 1 — close pre-deployment policy gaps

1. In Harbor, change the release retention rule to match the tags the workflow actually emits
   (plain semantic versions and their SHA-qualified companion tags), or change the workflow to retain
   the `v` prefix. Prefer changing retention so existing image naming remains stable.
2. Confirm `buildcache` remains explicitly retained and excluded from application deployment policy.
3. Confirm the exact latest digest is pullable on both k3s nodes after its Trivy scan passes.
4. Run `make test` locally and confirm the published image reports `dd67eb7` from `/api/version`.
5. Decide whether `magnus/btcprice` should remain public.

**Gate:** no Kubernetes manifests are merged until tests pass, the latest immutable image is approved
by Harbor, and both nodes can pull it.

## Phase 2 — represent the external btcdata dependency in Kubernetes

Create the following in `../deploy/apps/btcprice/`:

- `namespace.yaml`
- `btcdata-service.yaml`
- `btcdata-endpointslice.yaml`

Use namespace `btcprice`. Define a selectorless ClusterIP Service named `btcdata` on port `4102` and
an EndpointSlice with:

- label `kubernetes.io/service-name: btcdata`;
- address type `IPv4`;
- endpoint `100.75.8.131`;
- TCP port `4102`.

The application then uses only:

```text
BTCDATA_URL=http://btcdata:4102
```

Do not embed citadel's IP in the Deployment. This abstraction lets btcdata move into Kubernetes later
without changing btcprice.

Test from disposable pods pinned once to `srv1893094` and once to `citadel`:

- `http://btcdata:4102/api/health` returns 200;
- `/api/kraken-minute` and `/api/kraken-hour` return JSON;
- WebSocket connections to `/api/price/ws` and `/api/strike/ws` receive data.

**Gate:** all HTTP and WebSocket checks pass from both nodes.

## Phase 3 — add the btcprice workload to `magnus/deploy`

First synchronize the GitOps checkout:

```bash
git -C ../deploy pull --ff-only
git -C ../deploy switch main
```

Add this structure:

```text
apps/btcprice/
  namespace.yaml
  btcdata-service.yaml
  btcdata-endpointslice.yaml
  deployment.yaml
  service.yaml
  ingress.yaml
  pdb.yaml
  kustomization.yaml
infrastructure/image-automation/
  btcprice-image.yaml
```

Also update:

- `apps/kustomization.yaml` to include `btcprice`;
- `infrastructure/image-automation/kustomization.yaml` to include `btcprice-image.yaml`.

### Deployment specification

Use:

- image `harbor.pmagnus.cloud/btcprice/btcprice:<known-good-main-tag>`;
- Flux marker `# {"$imagepolicy": "flux-system:btcprice"}` on the image;
- two replicas and container port `4040`;
- `BTCDATA_URL=http://btcdata:4102`;
- `terminationGracePeriodSeconds: 30`;
- rolling update with `maxUnavailable: 1` and `maxSurge: 0`;
- initial resources of approximately `50m` CPU / `64Mi` memory requested and `250m` CPU / `256Mi`
  memory limited, then tune from observed metrics.

Guarantee one replica on each node with:

- required node affinity allowing only hostnames `srv1893094` and `citadel`;
- required pod anti-affinity on `kubernetes.io/hostname`, matching
  `app.kubernetes.io/name: btcprice`.

Hard anti-affinity plus two replicas and two eligible nodes gives one pod per node. `maxSurge: 0`
prevents rollout deadlock because a third pod cannot satisfy that anti-affinity.

Use probes:

- startup: `GET /api/health`;
- liveness: `GET /api/health`;
- readiness: `GET /api/ready`, with Kubernetes timeout longer than the application's three-second
  upstream timeout.

Never use dependency-aware readiness as liveness; a btcdata outage must not restart healthy
btcprice processes.

### Service, disruption budget, and ingress

- ClusterIP Service: port `80` to named container port `4040`.
- PDB: `minAvailable: 1`.
- Ingress class: `nginx`.
- Host: `btcprice.k8s.pmagnus.cloud`.
- Path: `/`, `Prefix`, backend `btcprice:80`.
- No Kubernetes TLS secret: TLS terminates at hostinger Traefik.
- Add ingress-nginx proxy read/send timeouts suitable for long-lived WebSockets (for example 3600
  seconds). WebSocket upgrade support itself is automatic.

## Phase 4 — add Flux image automation

Follow the existing `hello` convention in `../deploy/infrastructure/image-automation/`:

- `ImageRepository` named `btcprice` in `flux-system`;
- image `harbor.pmagnus.cloud/btcprice/btcprice`;
- existing pull-only secret `harbor-pull`;
- one-minute scan interval;
- `ImagePolicy` matching only plain branch tags:

```yaml
filterTags:
  pattern: '^main-(?P<ts>[0-9]{14})$'
  extract: '$ts'
policy:
  numerical:
    order: asc
```

This intentionally excludes SHA-qualified tags, releases, and `buildcache`. The existing
`ImageUpdateAutomation/apps` object will update files under `./apps`, commit to `magnus/deploy`, and
cause Flux to reconcile. CI receives no GitOps write key or kubeconfig.

Validate manifests before pushing:

```bash
kubectl kustomize ../deploy/apps/btcprice >/dev/null
kubectl kustomize ../deploy/apps >/dev/null
kubectl kustomize ../deploy/infrastructure/image-automation >/dev/null
```

Commit and push only to `magnus/deploy`; do not manually apply the resources. Watch Flux and verify
that all Kustomizations remain Ready.

## Phase 5 — enable ingress TLS

On hostinger, edit the file-provider configuration mounted at:

```text
/home/pmp/apps/netbird/traefik-dynamic.yaml
```

Add this entry under `http.routers.k8s-ingress.tls.domains`:

```yaml
- main: "btcprice.k8s.pmagnus.cloud"
```

Use `SUDO_HOSTINGER` from `../system/.env` only if privilege escalation is required. Back up the file
first and preserve permissions. Do **not** restart Traefik: its file provider hot-reloads, and a
restart can interrupt ACME and NetBird traffic.

Wait for issuance and verify externally:

```bash
curl --fail https://btcprice.k8s.pmagnus.cloud/api/health
curl --fail https://btcprice.k8s.pmagnus.cloud/api/ready
```

The certificate must be publicly trusted and named for `btcprice.k8s.pmagnus.cloud`; do not accept the
current self-signed `TRAEFIK DEFAULT CERT`.

## Phase 6 — acceptance and unattended deployment test

### Initial deployment

Confirm:

- both Flux application and infrastructure Kustomizations are Ready;
- `ImageRepository`, `ImagePolicy`, and `ImageUpdateAutomation` are Ready;
- exactly two btcprice pods are Ready, one on each node;
- Service endpoints contain both pods;
- `/`, CSS, favicon, chart APIs, health, readiness, and version endpoints work externally;
- `/ws/price` and `/ws/strike` remain connected and update through ingress;
- the running `/api/version` equals the expected image's embedded short SHA.

### Full pipeline test

Push a harmless visible change to `magnus/btcprice` and observe, without manual deployment actions:

1. Gitea queues the workflow on `anvil`.
2. Tests pass.
3. Harbor receives the timestamp and SHA-qualified tags.
4. Trivy approves the artifact.
5. Flux image reflection selects the new plain timestamp tag.
6. Flux commits the image change to `magnus/deploy`.
7. kustomize-controller rolls out the Deployment.
8. `/api/version` changes to the pushed source SHA.

### Failure tests

- Delete each pod in turn; one endpoint remains available and the missing node gets a replacement.
- During a rollout, verify browser WebSockets reconnect after old pods terminate.
- Temporarily make btcdata unreachable in a controlled window; readiness becomes false, liveness
  remains true, and no restart loop occurs.
- Verify a Harbor-rejected image cannot become a healthy rollout.
- Revert a GitOps image commit and confirm Flux restores the prior immutable image.

**Acceptance criterion:** a normal source push changes production with no SSH, Compose, or manual
`kubectl` deployment, while Git records the deployed image.

## Phase 7 — cut over from Compose

Keep `http://192.168.50.195:4041` running in parallel for at least 24 hours after all acceptance tests
pass. It must not own the new Kubernetes hostname.

Then on citadel:

1. Record the legacy image ID and effective environment for rollback without recording secrets.
2. Stop only `/home/pmp/apps/btcprice`; do not stop btcdata, Frankfurter, PostgreSQL, or remove
   `btc-network`.
3. Verify no clients still use port 4041.
4. Keep `deploy-citadel.sh`, `docker-compose.citadel.yml`, and `.env.citadel` for one defined rollback
   window, clearly marked legacy/manual-only.
5. After that window, remove the legacy deployment files and update `README.md`, `CLAUDE.md`, and
   `../system/infrastructure.md` so GitOps is the only production path.

## Rollback

Application rollback:

1. Suspend the btcprice `ImagePolicy` or image automation if it would immediately reselect the bad
   tag.
2. Revert the automated image commit in `magnus/deploy`, or commit the previous known-good immutable
   tag.
3. Let Flux reconcile and verify readiness/version.
4. Re-enable automation only after the bad tag is no longer selectable.

Migration rollback while Compose is retained:

1. Remove or suspend the btcprice Ingress through Git and let Flux reconcile.
2. Restart only the recorded btcprice Compose service on citadel.
3. Restore clients temporarily to port 4041.
4. Diagnose without deleting Harbor artifacts or rewriting Git history.

## Known limitation and follow-up

Two frontend replicas do not make the full system highly available: btcdata and its database remain
single-node dependencies on citadel. If citadel fails, the hostinger btcprice pod may serve static
content but readiness and live data fail. Migrate btcdata and decide PostgreSQL storage/availability
in a separate change; then replace the selectorless Service and EndpointSlice with a normal
Kubernetes Service.