btcprice/btc-price-deploy.md

btcprice GitOps migration plan

Target path

git push (magnus/btcprice)
  -> Gitea Actions on anvil
  -> Harbor image + Trivy scan
  -> Flux image policy commits magnus/deploy
  -> Flux reconciles k3s
  -> ingress-nginx
  -> hostinger Traefik TLS
  -> https://btcprice.k8s.pmagnus.cloud

Production deployment must not use docker compose, kubectl apply from CI, or a CI-held kubeconfig. The existing Compose instance on citadel is retained only during validation and as a short rollback path.

Access and secret handling

Use /Users/pmagnus/pi/system/.env for Gitea, Harbor, host and Cloudflare credentials. The file is mode 0600 and must remain outside Git.

set -a
. ../system/.env
set +a

Rules:

• never echo tokens/passwords or put them in command arguments;
• pass secrets via API headers, stdin, or temporary 0600 files;
• use the Harbor CI robot for image pushes, the existing cluster pull robot for pulls, and Harbor admin only for project policy changes;
• use the Gitea token for API administration, not from application or workflow code;
• do not copy ../system/.env into this repository, an image, a pod, or magnus/deploy.

Audited starting point (2026-09-15)

Already working:

• magnus/btcprice uses main; .gitea/workflows/build.yml tests and publishes immutable amd64 images from anvil.
• The latest Gitea run for commit dd67eb7 succeeded.
• Harbor project btcprice is private, scans on push, blocks High-or-worse vulnerable images, has a retention policy and weekly garbage collection. The latest image is scanned successfully: main-20260915160353 / main-20260915160353-dd67eb7.
• The repository has HARBOR_ROBOT_USER and HARBOR_ROBOT_TOKEN Actions secrets.
• The image exposes /api/health, dependency-aware /api/ready, and /api/version.
• Flux and all image automation controllers are healthy. The hello canary proves unattended image update, reconciliation, two-node scheduling, ingress, and TLS.
• k3s has two Ready amd64 nodes: srv1893094 and citadel.
• Both nodes already have Harbor pull credentials.
• ingress-nginx is managed by Flux and is reached through hostinger Traefik on NodePort 30080.
• btcdata still runs under Compose on citadel and answers on 100.75.8.131:4102.
• The old btcprice Compose container remains available on citadel port 4041.

Current gaps:

• magnus/deploy has no btcprice resources or image policy.
• btcprice.k8s.pmagnus.cloud currently reaches Traefik but receives its default certificate and an nginx 404; the TLS domain and Kubernetes Ingress do not exist yet.
• Local ../deploy can lag Flux's automated commits. Always run git pull --ff-only before editing.
• Harbor's release retention rule currently keeps v*, while the workflow strips the leading v (v1.2.3 becomes 1.2.3). Correct that mismatch before relying on release retention.
• The Gitea btcprice repository is currently public. Confirm that this is intentional; it does not expose the Actions secrets, but it does expose source and workflow definitions.

Phase 1 — close pre-deployment policy gaps

1. In Harbor, change the release retention rule to match the tags the workflow actually emits (plain semantic versions and their SHA-qualified companion tags), or change the workflow to retain the v prefix. Prefer changing retention so existing image naming remains stable.
2. Confirm buildcache remains explicitly retained and excluded from application deployment policy.
3. Confirm the exact latest digest is pullable on both k3s nodes after its Trivy scan passes.
4. Run make test locally and confirm the published image reports dd67eb7 from /api/version.
5. Decide whether magnus/btcprice should remain public.

Gate: no Kubernetes manifests are merged until tests pass, the latest immutable image is approved by Harbor, and both nodes can pull it.

Phase 2 — represent the external btcdata dependency in Kubernetes

Create the following in ../deploy/apps/btcprice/:

• namespace.yaml
• btcdata-service.yaml
• btcdata-endpointslice.yaml

Use namespace btcprice. Define a selectorless ClusterIP Service named btcdata on port 4102 and an EndpointSlice with:

• label kubernetes.io/service-name: btcdata;
• address type IPv4;
• endpoint 100.75.8.131;
• TCP port 4102.

The application then uses only:

BTCDATA_URL=http://btcdata:4102

Do not embed citadel's IP in the Deployment. This abstraction lets btcdata move into Kubernetes later without changing btcprice.

Test from disposable pods pinned once to srv1893094 and once to citadel:

• http://btcdata:4102/api/health returns 200;
• /api/kraken-minute and /api/kraken-hour return JSON;
• WebSocket connections to /api/price/ws and /api/strike/ws receive data.

Gate: all HTTP and WebSocket checks pass from both nodes.

Phase 3 — add the btcprice workload to magnus/deploy

First synchronize the GitOps checkout:

git -C ../deploy pull --ff-only
git -C ../deploy switch main

Add this structure:

apps/btcprice/
  namespace.yaml
  btcdata-service.yaml
  btcdata-endpointslice.yaml
  deployment.yaml
  service.yaml
  ingress.yaml
  pdb.yaml
  kustomization.yaml
infrastructure/image-automation/
  btcprice-image.yaml

Also update:

• apps/kustomization.yaml to include btcprice;
• infrastructure/image-automation/kustomization.yaml to include btcprice-image.yaml.

Deployment specification

Use:

• image harbor.pmagnus.cloud/btcprice/btcprice:<known-good-main-tag>;
• Flux marker # {"$imagepolicy": "flux-system:btcprice"} on the image;
• two replicas and container port 4040;
• BTCDATA_URL=http://btcdata:4102;
• terminationGracePeriodSeconds: 30;
• rolling update with maxUnavailable: 1 and maxSurge: 0;
• initial resources of approximately 50m CPU / 64Mi memory requested and 250m CPU / 256Mi memory limited, then tune from observed metrics.

Guarantee one replica on each node with:

• required node affinity allowing only hostnames srv1893094 and citadel;
• required pod anti-affinity on kubernetes.io/hostname, matching app.kubernetes.io/name: btcprice.

Hard anti-affinity plus two replicas and two eligible nodes gives one pod per node. maxSurge: 0 prevents rollout deadlock because a third pod cannot satisfy that anti-affinity.

Use probes:

• startup: GET /api/health;
• liveness: GET /api/health;
• readiness: GET /api/ready, with Kubernetes timeout longer than the application's three-second upstream timeout.

Never use dependency-aware readiness as liveness; a btcdata outage must not restart healthy btcprice processes.

Service, disruption budget, and ingress

• ClusterIP Service: port 80 to named container port 4040.
• PDB: minAvailable: 1.
• Ingress class: nginx.
• Host: btcprice.k8s.pmagnus.cloud.
• Path: /, Prefix, backend btcprice:80.
• No Kubernetes TLS secret: TLS terminates at hostinger Traefik.
• Add ingress-nginx proxy read/send timeouts suitable for long-lived WebSockets (for example 3600 seconds). WebSocket upgrade support itself is automatic.

Phase 4 — add Flux image automation

Follow the existing hello convention in ../deploy/infrastructure/image-automation/:

• ImageRepository named btcprice in flux-system;
• image harbor.pmagnus.cloud/btcprice/btcprice;
• existing pull-only secret harbor-pull;
• one-minute scan interval;
• ImagePolicy matching only plain branch tags:

filterTags:
  pattern: '^main-(?P<ts>[0-9]{14})$'
  extract: '$ts'
policy:
  numerical:
    order: asc

This intentionally excludes SHA-qualified tags, releases, and buildcache. The existing ImageUpdateAutomation/apps object will update files under ./apps, commit to magnus/deploy, and cause Flux to reconcile. CI receives no GitOps write key or kubeconfig.

Validate manifests before pushing:

kubectl kustomize ../deploy/apps/btcprice >/dev/null
kubectl kustomize ../deploy/apps >/dev/null
kubectl kustomize ../deploy/infrastructure/image-automation >/dev/null

Commit and push only to magnus/deploy; do not manually apply the resources. Watch Flux and verify that all Kustomizations remain Ready.

Phase 5 — enable ingress TLS

On hostinger, edit the file-provider configuration mounted at:

/home/pmp/apps/netbird/traefik-dynamic.yaml

Add this entry under http.routers.k8s-ingress.tls.domains:

- main: "btcprice.k8s.pmagnus.cloud"

Use SUDO_HOSTINGER from ../system/.env only if privilege escalation is required. Back up the file first and preserve permissions. Do not restart Traefik: its file provider hot-reloads, and a restart can interrupt ACME and NetBird traffic.

Wait for issuance and verify externally:

curl --fail https://btcprice.k8s.pmagnus.cloud/api/health
curl --fail https://btcprice.k8s.pmagnus.cloud/api/ready

The certificate must be publicly trusted and named for btcprice.k8s.pmagnus.cloud; do not accept the current self-signed TRAEFIK DEFAULT CERT.

Phase 6 — acceptance and unattended deployment test

Initial deployment

Confirm:

• both Flux application and infrastructure Kustomizations are Ready;
• ImageRepository, ImagePolicy, and ImageUpdateAutomation are Ready;
• exactly two btcprice pods are Ready, one on each node;
• Service endpoints contain both pods;
• /, CSS, favicon, chart APIs, health, readiness, and version endpoints work externally;
• /ws/price and /ws/strike remain connected and update through ingress;
• the running /api/version equals the expected image's embedded short SHA.

Full pipeline test

Push a harmless visible change to magnus/btcprice and observe, without manual deployment actions:

1. Gitea queues the workflow on anvil.
2. Tests pass.
3. Harbor receives the timestamp and SHA-qualified tags.
4. Trivy approves the artifact.
5. Flux image reflection selects the new plain timestamp tag.
6. Flux commits the image change to magnus/deploy.
7. kustomize-controller rolls out the Deployment.
8. /api/version changes to the pushed source SHA.

Failure tests

• Delete each pod in turn; one endpoint remains available and the missing node gets a replacement.
• During a rollout, verify browser WebSockets reconnect after old pods terminate.
• Temporarily make btcdata unreachable in a controlled window; readiness becomes false, liveness remains true, and no restart loop occurs.
• Verify a Harbor-rejected image cannot become a healthy rollout.
• Revert a GitOps image commit and confirm Flux restores the prior immutable image.

Acceptance criterion: a normal source push changes production with no SSH, Compose, or manual kubectl deployment, while Git records the deployed image.

Phase 7 — cut over from Compose

Keep http://192.168.50.195:4041 running in parallel for at least 24 hours after all acceptance tests pass. It must not own the new Kubernetes hostname.

Then on citadel:

1. Record the legacy image ID and effective environment for rollback without recording secrets.
2. Stop only /home/pmp/apps/btcprice; do not stop btcdata, Frankfurter, PostgreSQL, or remove btc-network.
3. Verify no clients still use port 4041.
4. Keep deploy-citadel.sh, docker-compose.citadel.yml, and .env.citadel for one defined rollback window, clearly marked legacy/manual-only.
5. After that window, remove the legacy deployment files and update README.md, CLAUDE.md, and ../system/infrastructure.md so GitOps is the only production path.

Rollback

Application rollback:

1. Suspend the btcprice ImagePolicy or image automation if it would immediately reselect the bad tag.
2. Revert the automated image commit in magnus/deploy, or commit the previous known-good immutable tag.
3. Let Flux reconcile and verify readiness/version.
4. Re-enable automation only after the bad tag is no longer selectable.

Migration rollback while Compose is retained:

1. Remove or suspend the btcprice Ingress through Git and let Flux reconcile.
2. Restart only the recorded btcprice Compose service on citadel.
3. Restore clients temporarily to port 4041.
4. Diagnose without deleting Harbor artifacts or rewriting Git history.

Known limitation and follow-up

Two frontend replicas do not make the full system highly available: btcdata and its database remain single-node dependencies on citadel. If citadel fails, the hostinger btcprice pod may serve static content but readiness and live data fail. Migrate btcdata and decide PostgreSQL storage/availability in a separate change; then replace the selectorless Service and EndpointSlice with a normal Kubernetes Service.