magnus/btcprice
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0470862cb4fa7c39754effe933a6b7bdc87d2fcf
btcprice/btc-price-deploy.md
T

20 KiB
Raw Blame History

btcprice CI/CD and Kubernetes deployment plan

Goal

Replace the current workstation-driven deployment:

local docker build -> docker save over SSH -> Docker Compose on citadel

with the CI/CD path defined in /Users/pmagnus/pi/system/cicd-plan.md:

git push to Gitea
  -> Gitea Actions runner `anvil`
  -> build and push to Harbor
  -> Flux image automation commits a tag update to `magnus/deploy`
  -> Flux reconciles k3s
  -> ingress-nginx serves btcprice through hostinger's Traefik

The final Kubernetes Deployment must run two replicas: exactly one on srv1893094 (hostinger) and one on citadel. The existing Compose deployment remains available during migration and is removed only after the Kubernetes service is proven.

Proposed public URL: https://btcprice.k8s.pmagnus.cloud.

Current state and constraints

  • Source is already in Gitea at magnus/btcprice and currently uses the master branch.
  • deploy-citadel.sh builds locally, copies the image over SSH and starts docker-compose.citadel.yml on citadel.
  • The current container listens on port 4040; the Compose deployment publishes it as 192.168.50.195:4041.
  • btcprice is stateless and needs no PVC.
  • btcprice depends on btcdata for HTTP and WebSocket upstreams.
  • btcdata remains a Compose service on citadel and is published on 0.0.0.0:4102.
  • Both cluster nodes can currently reach btcdata at http://100.75.8.131:4102; this was checked from hostinger and citadel and returned HTTP 200 from /api/health.
  • The k3s cluster already consists of:
    • srv1893094: server/control plane on hostinger, overlay IP 100.75.252.111;
    • citadel: agent, overlay IP 100.75.8.131.
  • Both nodes can pull private Harbor images through the node-level robot$cluster-pull credential.
  • ingress-nginx is already exposed through hostinger's Traefik at *.k8s.pmagnus.cloud.
  • Gitea Actions, anvil, Harbor builds and scans work, but Flux/CD is not yet installed according to the system plan.

Important availability boundary

Two btcprice replicas do not make the complete application highly available while btcdata and its PostgreSQL database remain only on citadel. If citadel fails, the hostinger btcprice pod can continue serving static UI and its own health endpoint, but live prices and charts will fail. Migrating or making btcdata highly available is a separate follow-up project.

Target design

Internet
   |
   v
hostinger Traefik (TLS)
   |
   v
hostinger ingress-nginx :30080
   |
   v
ClusterIP Service btcprice:80
   |-------------------------------|
   v                               v
btcprice pod                   btcprice pod
srv1893094                     citadel
   |                               |
   +---- Service/EndpointSlice ----+
                |
                v
      btcdata at 100.75.8.131:4102
      (temporary external dependency)

Ingress-nginx supports WebSocket upgrades without a btcprice-specific WebSocket annotation. Connections are long-lived and may terminate on either replica; btcprice stores no session state, so no sticky session is required.

Phase 0 — finish and harden the shared CI/CD road

Do this before onboarding btcprice. Do not bypass it with a workflow that contains kubectl credentials.

  1. Complete /Users/pmagnus/pi/system/cicd-plan.md Phase 3:
    • create private Gitea repo magnus/deploy;
    • bootstrap Flux against clusters/k3s;
    • install source-controller, kustomize-controller, image-reflector-controller and image-automation-controller;
    • bring the existing ingress-nginx resources under Flux ownership without deleting the live controller;
    • prove the full path using magnus/hello.
  2. Meet the canary success criterion first: a push to magnus/hello must change the response from https://hello.k8s.pmagnus.cloud with no manual cluster action.
  3. Configure Harbor before btcprice starts producing per-commit tags:
    • scan on push;
    • prevent pulls of vulnerable images at the agreed severity, initially High;
    • retain the last 10 main-* builds and all v* releases;
    • exclude the buildcache tag from retention;
    • delete untagged artifacts and schedule weekly garbage collection.
  4. Prefer creating the planned Docker Hub proxy-cache project before repeated builds. Update the Dockerfile base images to Harbor proxy-cache names only after the cache is verified.
  5. Add 100.75.177.123 harbor.pmagnus.cloud persistently on citadel as described in the system plan, so image pulls do not depend on Technitium running on the same machine.
  6. Resolve the shared infrastructure safety items before making deployment automatic:
    • back up Harbor, anvil, Gitea/deploy and the k3s datastore;
    • change /opt/harbor/harbor.yml to mode 0600;
    • remove the duplicate ACME issuer for harbor.pmagnus.cloud.

Gate: do not proceed until Flux reconciliation, image reflection/automation, private image pulls on both nodes and the canary ingress all work.

Phase 1 — make btcprice deployment-ready

1.1 Keep runtime configuration external

The Kubernetes manifest, not the image, must set:

BTCDATA_URL=http://btcdata:4102

Remove or correct the stale Dockerfile default http://btcdata:4101. A production image should not silently point to the wrong port. Retain http://localhost:4100 as the application's local-development default if desired.

1.2 Add dependency-aware readiness

GET /api/health currently proves only that FastAPI is running. Keep it as the liveness endpoint and add a readiness endpoint, for example GET /api/ready, which performs a short, bounded request to ${BTCDATA_URL}/api/health:

  • return 200 only when btcdata answers successfully;
  • use a 2–3 second timeout;
  • return 503 without exposing internal exception details;
  • do not use readiness for liveness, otherwise a btcdata outage would restart healthy btcprice pods continuously.

Add tests for the ready and unavailable cases. Add pytest and FastAPI's test client dependencies to a development/test requirements file rather than the production image if practical.

1.3 Validate container behavior

Before enabling CI:

  1. Build the image locally.
  2. Run it with BTCDATA_URL=http://100.75.8.131:4102.
  3. Verify:
    • /api/health returns 200;
    • /api/ready returns 200;
    • both chart proxy endpoints return JSON;
    • /ws/price and /ws/strike receive updates;
    • the process exits cleanly on SIGTERM within the planned pod termination grace period.
  4. Optionally add a non-root runtime user. If done, confirm static files remain readable and Uvicorn can bind to port 4040.

1.4 Branch decision

The shared workflow template watches main, while this repo currently uses master. Choose one explicitly:

  • preferred: rename the default branch to main in Git and Gitea, then update local tracking; or
  • keep master and set the workflow trigger to branches: [master].

Do not copy the template unchanged or pushes will not trigger builds. The remaining examples assume main.

Phase 2 — Harbor and Gitea Actions build pipeline

2.1 Harbor repository and credentials

Create a private Harbor project btcprice and use image:

harbor.pmagnus.cloud/btcprice/btcprice

Create a project-scoped robot such as robot$btcprice+ci with only the permissions needed to push and pull artifacts in this project. Verify that it cannot access unrelated projects or perform administrative deletion.

Store its one-time secret in the btcprice Gitea repository as:

  • HARBOR_ROBOT_USER
  • HARBOR_ROBOT_TOKEN

Do not use Harbor admin credentials and do not place the token in the runner environment.

The existing system-level robot$cluster-pull already covers future projects through namespace: "*"; verify a private pull from the new project on both nodes before relying on it.

2.2 Add .gitea/workflows/build.yml

Base it on the proven magnus/hello workflow, with these btcprice-specific values:

env:
  REGISTRY: harbor.pmagnus.cloud
  IMAGE: harbor.pmagnus.cloud/btcprice/btcprice

Required behavior:

  1. Trigger on pushes to main and tags matching v*.
  2. Check out source.
  3. Run automated tests before publishing.
  4. Produce immutable tags:
    • branch: main-<UTC YYYYMMDDHHMMSS> and main-<timestamp>-<short SHA>;
    • release: version without the leading v, plus a SHA-qualified tag.
  5. Log in with the project robot.
  6. Build for linux/amd64; both current cluster nodes are amd64.
  7. Push through BuildKit with registry cache:
    • cache-from: type=registry,ref=...:buildcache;
    • cache-to: type=registry,ref=...:buildcache,mode=max.
  8. Preserve the working anvil networking workaround from the canary pipeline: use driver-opts: network=host in docker/setup-buildx-action, because anvil is not a NetBird peer. Its runner config must retain the Harbor --add-host option.
  9. Do not provide a kubeconfig or deploy from the workflow.

The Flux policy must match only the plain timestamp tag, not the second SHA-qualified tag. Use a pattern equivalent to:

^main-(?P<ts>[0-9]{14})$

and numerical ordering by ts.

2.3 Prove the build half independently

Push a harmless commit and verify:

  • the Gitea Actions run completes on anvil;
  • Harbor contains both immutable tags and buildcache;
  • Trivy scan completes;
  • the manifest architecture is amd64;
  • each k3s node can independently pull the exact digest;
  • no latest tag is required for deployment.

Stop and fix the build path before creating application manifests if any check fails.

Phase 3 — describe btcdata temporarily inside Kubernetes

Do not put the overlay IP directly in the btcprice Deployment. Add an abstraction in magnus/deploy, under the btcprice namespace:

  • a selectorless Service named btcdata, port 4102;
  • an EndpointSlice for that Service with endpoint 100.75.8.131, port 4102.

This gives btcprice the stable URL http://btcdata:4102. When btcdata is later migrated into Kubernetes, replace the EndpointSlice with normal pod selectors without changing btcprice.

Before committing it, verify from a temporary pod pinned to each node that:

http://btcdata:4102/api/health

returns 200 and that a WebSocket connection can be established.

Security follow-up: btcdata currently publishes 4102 on all interfaces. Limit access to the NetBird/cluster path with a tested firewall rule or a safe Docker binding. Do not change its binding in a way that races NetBird startup or breaks the existing Compose btc-network connection.

Phase 4 — add btcprice to the GitOps repository

Create this structure in magnus/deploy:

apps/btcprice/
  namespace.yaml
  btcdata-service.yaml
  btcdata-endpointslice.yaml
  deployment.yaml
  service.yaml
  ingress.yaml
  pdb.yaml
  image-repository.yaml
  image-policy.yaml
  kustomization.yaml

Reference it from the cluster's applications Kustomization.

4.1 Deployment

Use:

  • replicas: 2;
  • image harbor.pmagnus.cloud/btcprice/btcprice:<initial-known-good-tag>;
  • Flux image-policy marker on the image field;
  • container port 4040;
  • BTCDATA_URL=http://btcdata:4102;
  • conservative CPU/memory requests and limits based on a measured Compose container, then tune from metrics;
  • terminationGracePeriodSeconds sufficient to close WebSockets cleanly;
  • rolling strategy maxUnavailable: 1, maxSurge: 0.

Guarantee one replica per cluster node with both:

  1. node affinity restricting eligible hosts to srv1893094 and citadel;
  2. required pod anti-affinity on kubernetes.io/hostname for pods with app.kubernetes.io/name=btcprice.

Hard anti-affinity is intentional: with exactly two eligible nodes and two replicas it guarantees one pod on each. maxSurge: 0 avoids a rollout deadlock, because a third surge pod cannot satisfy hard anti-affinity.

Add probes:

  • startup: /api/health with enough allowance for startup/image extraction;
  • liveness: /api/health;
  • readiness: /api/ready.

Add standard labels and make selectors immutable and consistent.

4.2 Pod disruption budget

Create a PDB with minAvailable: 1. This protects voluntary maintenance from removing both replicas simultaneously. It does not make btcdata highly available and does not prevent an involuntary node failure.

4.3 Service and ingress

Create a ClusterIP Service mapping port 80 to container port 4040. Create an ingress with:

  • ingress class nginx;
  • host btcprice.k8s.pmagnus.cloud;
  • path /, prefix match;
  • backend btcprice:80;
  • no in-cluster TLS secret, because hostinger Traefik terminates TLS.

Use ingress-nginx timeouts long enough for the two WebSocket endpoints if the controller defaults or shared configuration are shorter than intended. Do not add session affinity unless testing demonstrates a real need.

4.4 Flux image automation

Add:

  • ImageRepository for harbor.pmagnus.cloud/btcprice/btcprice;
  • registry authentication using Flux's pull-only Harbor secret, not the CI push robot;
  • ImagePolicy selecting only main-<14 digit timestamp> tags numerically;
  • the Deployment image marker consumed by the existing ImageUpdateAutomation.

Flux should commit the selected tag to magnus/deploy. The committed manifest remains the auditable source of truth, and rollback is a Git revert to an earlier immutable tag.

Phase 5 — expose TLS and deploy alongside Compose

  1. Add btcprice.k8s.pmagnus.cloud as a tls.domains entry on hostinger's existing k8s-ingress Traefik router. TLS-ALPN-01 cannot issue the wildcard, so this per-app entry is required until DNS-01 is introduced.
  2. Edit traefik-dynamic.yaml once and allow its file provider to hot-reload. Do not restart Traefik; the system record shows that restart can interrupt ACME validation and NetBird traffic.
  3. Confirm the certificate is valid externally before treating application errors as Kubernetes errors.
  4. Commit the btcprice manifests with a known-good image tag.
  5. Confirm Flux reconciliation rather than applying the resources manually.
  6. Keep the Compose service on 192.168.50.195:4041 running during this phase. Both deployments can safely use the same stateless btcdata API.

Phase 6 — acceptance tests

Placement and rollout

  • Exactly two Ready btcprice pods exist.
  • One is on srv1893094; one is on citadel.
  • Delete each pod in turn and verify it is recreated on the same missing node.
  • Push a source change and verify the full unattended chain:
    1. Gitea queues the build;
    2. anvil tests, builds and pushes;
    3. Harbor scans the image;
    4. Flux selects the new timestamp tag;
    5. Flux commits the tag change;
    6. kustomize-controller rolls out the Deployment;
    7. the URL serves the new version.

Expose the build SHA in application output or a small version endpoint so this can be verified without guessing from pod metadata.

Functional checks

From outside the cluster and over a sustained test period:

  • https://btcprice.k8s.pmagnus.cloud/ loads with a valid certificate;
  • CSS, favicon and external JavaScript assets load;
  • /api/health and /api/ready return 200;
  • minute and hourly chart requests return data;
  • /ws/price and /ws/strike stay connected and update;
  • ingress logs show successful WebSocket upgrades;
  • browser refreshes are served successfully while one pod is deleted;
  • a rollout terminates old WebSocket connections cleanly and browser reconnection recovers;
  • no image pull, DNS, MTU or cross-node timeout errors appear.

Failure checks

  • Stop one btcprice pod: the site remains usable through the other.
  • Temporarily make btcdata unreachable in a controlled window: readiness becomes false without a liveness restart loop, and recovers when btcdata returns.
  • Attempt to schedule both replicas on one node: hard anti-affinity must leave the extra replica Pending rather than violate placement.
  • Verify a vulnerable/rejected image cannot roll out and the previous ReplicaSet remains available.

Acceptance criterion: a push changes the running application with no human deployment step, and the service remains available with either btcprice pod absent.

Phase 7 — cut over and retire the old deployment

After at least 24 hours of successful Kubernetes operation:

  1. Record the currently running Compose image and configuration for rollback.

  2. Stop only the btcprice Compose project on citadel:

    cd /home/pmp/apps/btcprice
docker compose down

    Do not stop btcdata, frankfurter or remove the shared btc-network.

  3. Confirm port 4041 is no longer needed. If LAN clients still use http://192.168.50.195:4041, move them to the HTTPS Kubernetes hostname before closing it.

  4. Mark these as legacy/manual-only, but keep them for one rollback window:

    • deploy-citadel.sh;
    • docker-compose.citadel.yml;
    • .env.citadel.

  5. After the rollback window, remove the legacy deployment files or move them to a clearly documented archive. Update README.md and CLAUDE.md so git push/Flux is the only documented production deployment path.

  6. Update /Users/pmagnus/pi/system/infrastructure.md to move btcprice from citadel's Compose list to k3s workloads and record its two-node placement and public URL.

Do not have Compose and Flux both claim the production endpoint. Compose is only a temporary parallel validation and emergency rollback path.

Rollback

Application release rollback

  1. Revert the Flux image-automation commit in magnus/deploy, or commit the last known-good immutable tag.
  2. Suspend btcprice image automation temporarily if it would immediately select the bad newer tag again.
  3. Let Flux reconcile; do not use an unrecorded kubectl set image except for immediate incident containment, and commit any emergency change back to Git.
  4. Re-enable automation only after the bad image/tag is excluded or replaced.

Migration rollback

While the Compose rollback window is open:

  1. Remove or disable the btcprice ingress in Git and let Flux reconcile.
  2. Restart /home/pmp/apps/btcprice on citadel with the recorded Compose image/configuration.
  3. Restore clients to http://192.168.50.195:4041 if necessary.
  4. Diagnose without deleting Harbor tags or Git history.

Shared dependency rollback

If the selectorless btcdata Service/EndpointSlice is the fault, revert those manifests. Do not move btcdata itself during the same change window as btcprice; that would combine two independent migrations and obscure failures.

Follow-up work

  1. Migrate btcdata and Frankfurter to Kubernetes in a separate plan.
  2. Decide how to handle PostgreSQL before migrating btcdata; the current database is native and single-node on citadel.
  3. Replace the manual EndpointSlice when btcdata gets a Kubernetes Service.
  4. Add monitoring for:
    • both btcprice replicas and their node placement;
    • Flux reconciliation and image automation failures;
    • Harbor scan/retention/GC;
    • ingress 5xx and WebSocket disconnect rate;
    • btcdata readiness and latency;
    • Harbor certificate renewal.
  5. Consider Cloudflare DNS-01 for a real *.k8s.pmagnus.cloud certificate, removing per-app Traefik certificate entries.
  6. Enroll anvil in NetBird if desired, then remove the runner --add-host and BuildKit host-network workarounds only after a successful test build.

Implementation checklist

  • Flux/CD and magnus/hello canary completed
  • Harbor retention, GC, scanning and vulnerability policy configured
  • Shared infrastructure backups verified
  • btcprice branch trigger resolved (main versus master)
  • stale Dockerfile btcdata port removed/corrected
  • dependency-aware /api/ready and tests added
  • local container and WebSocket smoke tests passed
  • Harbor btcprice project and least-privilege CI robot created
  • Gitea repository secrets added
  • Gitea Actions build/test/push workflow added
  • scanned immutable image proven pullable on both nodes
  • selectorless btcdata Service and EndpointSlice tested from both nodes
  • two-replica Deployment with required anti-affinity committed
  • Service, ingress, PDB and probes committed
  • Flux ImageRepository/ImagePolicy/automation marker committed
  • Traefik TLS domain added by hot reload, without restart
  • one Ready pod verified on hostinger and one on citadel
  • end-to-end push-to-production test passed
  • functional, node-failure and rollback tests passed
  • 24-hour parallel validation completed
  • old btcprice Compose service stopped without stopping btcdata
  • project and system documentation updated
Reference in New Issue View Git Blame Copy Permalink