magnus/btcprice
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare btcprice for Kubernetes deployment

Browse Source
This commit is contained in:
magnus
2026-09-15 17:12:25 +02:00
parent dda00da77a
commit 573eba2d86
8 changed files with 619 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
btc-price-deploy.md
+466
View File
@@ -0,0 +1,466 @@
# btcprice CI/CD and Kubernetes deployment plan
## Goal
Replace the current workstation-driven deployment:
```text
local docker build -> docker save over SSH -> Docker Compose on citadel
```
with the CI/CD path defined in `/Users/pmagnus/pi/system/cicd-plan.md`:
```text
git push to Gitea
-> Gitea Actions runner `anvil`
-> build and push to Harbor
-> Flux image automation commits a tag update to `magnus/deploy`
-> Flux reconciles k3s
-> ingress-nginx serves btcprice through hostinger's Traefik
```
The final Kubernetes Deployment must run **two replicas: exactly one on `srv1893094` (hostinger) and one on `citadel`**. The existing Compose deployment remains available during migration and is removed only after the Kubernetes service is proven.
Proposed public URL: **`https://btcprice.k8s.pmagnus.cloud`**.
---
## Current state and constraints
- Source is already in Gitea at `magnus/btcprice` and currently uses the `master` branch.
- `deploy-citadel.sh` builds locally, copies the image over SSH and starts `docker-compose.citadel.yml` on citadel.
- The current container listens on port `4040`; the Compose deployment publishes it as `192.168.50.195:4041`.
- btcprice is stateless and needs no PVC.
- btcprice depends on btcdata for HTTP and WebSocket upstreams.
- btcdata remains a Compose service on citadel and is published on `0.0.0.0:4102`.
- Both cluster nodes can currently reach btcdata at `http://100.75.8.131:4102`; this was checked from hostinger and citadel and returned HTTP 200 from `/api/health`.
- The k3s cluster already consists of:
- `srv1893094`: server/control plane on hostinger, overlay IP `100.75.252.111`;
- `citadel`: agent, overlay IP `100.75.8.131`.
- Both nodes can pull private Harbor images through the node-level `robot$cluster-pull` credential.
- ingress-nginx is already exposed through hostinger's Traefik at `*.k8s.pmagnus.cloud`.
- Gitea Actions, `anvil`, Harbor builds and scans work, but Flux/CD is not yet installed according to the system plan.
### Important availability boundary
Two btcprice replicas do not make the complete application highly available while btcdata and its PostgreSQL database remain only on citadel. If citadel fails, the hostinger btcprice pod can continue serving static UI and its own health endpoint, but live prices and charts will fail. Migrating or making btcdata highly available is a separate follow-up project.
---
## Target design
```text
Internet
|
v
hostinger Traefik (TLS)
|
v
hostinger ingress-nginx :30080
|
v
ClusterIP Service btcprice:80
|-------------------------------|
v v
btcprice pod btcprice pod
srv1893094 citadel
| |
+---- Service/EndpointSlice ----+
|
v
btcdata at 100.75.8.131:4102
(temporary external dependency)
```
Ingress-nginx supports WebSocket upgrades without a btcprice-specific WebSocket annotation. Connections are long-lived and may terminate on either replica; btcprice stores no session state, so no sticky session is required.
---
## Phase 0 — finish and harden the shared CI/CD road
Do this before onboarding btcprice. Do not bypass it with a workflow that contains `kubectl` credentials.
1. Complete `/Users/pmagnus/pi/system/cicd-plan.md` Phase 3:
- create private Gitea repo `magnus/deploy`;
- bootstrap Flux against `clusters/k3s`;
- install `source-controller`, `kustomize-controller`, `image-reflector-controller` and `image-automation-controller`;
- bring the existing ingress-nginx resources under Flux ownership without deleting the live controller;
- prove the full path using `magnus/hello`.
2. Meet the canary success criterion first: a push to `magnus/hello` must change the response from `https://hello.k8s.pmagnus.cloud` with no manual cluster action.
3. Configure Harbor before btcprice starts producing per-commit tags:
- scan on push;
- prevent pulls of vulnerable images at the agreed severity, initially High;
- retain the last 10 `main-*` builds and all `v*` releases;
- exclude the `buildcache` tag from retention;
- delete untagged artifacts and schedule weekly garbage collection.
4. Prefer creating the planned Docker Hub proxy-cache project before repeated builds. Update the Dockerfile base images to Harbor proxy-cache names only after the cache is verified.
5. Add `100.75.177.123 harbor.pmagnus.cloud` persistently on citadel as described in the system plan, so image pulls do not depend on Technitium running on the same machine.
6. Resolve the shared infrastructure safety items before making deployment automatic:
- back up Harbor, `anvil`, Gitea/deploy and the k3s datastore;
- change `/opt/harbor/harbor.yml` to mode `0600`;
- remove the duplicate ACME issuer for `harbor.pmagnus.cloud`.
**Gate:** do not proceed until Flux reconciliation, image reflection/automation, private image pulls on both nodes and the canary ingress all work.
---
## Phase 1 — make btcprice deployment-ready
### 1.1 Keep runtime configuration external
The Kubernetes manifest, not the image, must set:
```text
BTCDATA_URL=http://btcdata:4102
```
Remove or correct the stale Dockerfile default `http://btcdata:4101`. A production image should not silently point to the wrong port. Retain `http://localhost:4100` as the application's local-development default if desired.
### 1.2 Add dependency-aware readiness
`GET /api/health` currently proves only that FastAPI is running. Keep it as the liveness endpoint and add a readiness endpoint, for example `GET /api/ready`, which performs a short, bounded request to `${BTCDATA_URL}/api/health`:
- return 200 only when btcdata answers successfully;
- use a 2–3 second timeout;
- return 503 without exposing internal exception details;
- do not use readiness for liveness, otherwise a btcdata outage would restart healthy btcprice pods continuously.
Add tests for the ready and unavailable cases. Add `pytest` and FastAPI's test client dependencies to a development/test requirements file rather than the production image if practical.
### 1.3 Validate container behavior
Before enabling CI:
1. Build the image locally.
2. Run it with `BTCDATA_URL=http://100.75.8.131:4102`.
3. Verify:
- `/api/health` returns 200;
- `/api/ready` returns 200;
- both chart proxy endpoints return JSON;
- `/ws/price` and `/ws/strike` receive updates;
- the process exits cleanly on SIGTERM within the planned pod termination grace period.
4. Optionally add a non-root runtime user. If done, confirm static files remain readable and Uvicorn can bind to port 4040.
### 1.4 Branch decision
The shared workflow template watches `main`, while this repo currently uses `master`. Choose one explicitly:
- preferred: rename the default branch to `main` in Git and Gitea, then update local tracking; or
- keep `master` and set the workflow trigger to `branches: [master]`.
Do not copy the template unchanged or pushes will not trigger builds. The remaining examples assume `main`.
---
## Phase 2 — Harbor and Gitea Actions build pipeline
### 2.1 Harbor repository and credentials
Create a private Harbor project `btcprice` and use image:
```text
harbor.pmagnus.cloud/btcprice/btcprice
```
Create a project-scoped robot such as `robot$btcprice+ci` with only the permissions needed to push and pull artifacts in this project. Verify that it cannot access unrelated projects or perform administrative deletion.
Store its one-time secret in the **btcprice Gitea repository** as:
- `HARBOR_ROBOT_USER`
- `HARBOR_ROBOT_TOKEN`
Do not use Harbor admin credentials and do not place the token in the runner environment.
The existing system-level `robot$cluster-pull` already covers future projects through `namespace: "*"`; verify a private pull from the new project on both nodes before relying on it.
### 2.2 Add `.gitea/workflows/build.yml`
Base it on the proven `magnus/hello` workflow, with these btcprice-specific values:
```yaml
env:
REGISTRY: harbor.pmagnus.cloud
IMAGE: harbor.pmagnus.cloud/btcprice/btcprice
```
Required behavior:
1. Trigger on pushes to `main` and tags matching `v*`.
2. Check out source.
3. Run automated tests before publishing.
4. Produce immutable tags:
- branch: `main-<UTC YYYYMMDDHHMMSS>` and `main-<timestamp>-<short SHA>`;
- release: version without the leading `v`, plus a SHA-qualified tag.
5. Log in with the project robot.
6. Build for `linux/amd64`; both current cluster nodes are amd64.
7. Push through BuildKit with registry cache:
- `cache-from: type=registry,ref=...:buildcache`;
- `cache-to: type=registry,ref=...:buildcache,mode=max`.
8. Preserve the working `anvil` networking workaround from the canary pipeline: use `driver-opts: network=host` in `docker/setup-buildx-action`, because `anvil` is not a NetBird peer. Its runner config must retain the Harbor `--add-host` option.
9. Do not provide a kubeconfig or deploy from the workflow.
The Flux policy must match only the plain timestamp tag, not the second SHA-qualified tag. Use a pattern equivalent to:
```text
^main-(?P<ts>[0-9]{14})$
```
and numerical ordering by `ts`.
### 2.3 Prove the build half independently
Push a harmless commit and verify:
- the Gitea Actions run completes on `anvil`;
- Harbor contains both immutable tags and `buildcache`;
- Trivy scan completes;
- the manifest architecture is amd64;
- each k3s node can independently pull the exact digest;
- no `latest` tag is required for deployment.
Stop and fix the build path before creating application manifests if any check fails.
---
## Phase 3 — describe btcdata temporarily inside Kubernetes
Do not put the overlay IP directly in the btcprice Deployment. Add an abstraction in `magnus/deploy`, under the btcprice namespace:
- a selectorless Service named `btcdata`, port `4102`;
- an EndpointSlice for that Service with endpoint `100.75.8.131`, port `4102`.
This gives btcprice the stable URL `http://btcdata:4102`. When btcdata is later migrated into Kubernetes, replace the EndpointSlice with normal pod selectors without changing btcprice.
Before committing it, verify from a temporary pod pinned to **each node** that:
```text
http://btcdata:4102/api/health
```
returns 200 and that a WebSocket connection can be established.
Security follow-up: btcdata currently publishes `4102` on all interfaces. Limit access to the NetBird/cluster path with a tested firewall rule or a safe Docker binding. Do not change its binding in a way that races NetBird startup or breaks the existing Compose `btc-network` connection.
---
## Phase 4 — add btcprice to the GitOps repository
Create this structure in `magnus/deploy`:
```text
apps/btcprice/
namespace.yaml
btcdata-service.yaml
btcdata-endpointslice.yaml
deployment.yaml
service.yaml
ingress.yaml
pdb.yaml
image-repository.yaml
image-policy.yaml
kustomization.yaml
```
Reference it from the cluster's applications Kustomization.
### 4.1 Deployment
Use:
- `replicas: 2`;
- image `harbor.pmagnus.cloud/btcprice/btcprice:<initial-known-good-tag>`;
- Flux image-policy marker on the image field;
- container port `4040`;
- `BTCDATA_URL=http://btcdata:4102`;
- conservative CPU/memory requests and limits based on a measured Compose container, then tune from metrics;
- `terminationGracePeriodSeconds` sufficient to close WebSockets cleanly;
- rolling strategy `maxUnavailable: 1`, `maxSurge: 0`.
Guarantee one replica per cluster node with both:
1. node affinity restricting eligible hosts to `srv1893094` and `citadel`;
2. **required** pod anti-affinity on `kubernetes.io/hostname` for pods with `app.kubernetes.io/name=btcprice`.
Hard anti-affinity is intentional: with exactly two eligible nodes and two replicas it guarantees one pod on each. `maxSurge: 0` avoids a rollout deadlock, because a third surge pod cannot satisfy hard anti-affinity.
Add probes:
- startup: `/api/health` with enough allowance for startup/image extraction;
- liveness: `/api/health`;
- readiness: `/api/ready`.
Add standard labels and make selectors immutable and consistent.
### 4.2 Pod disruption budget
Create a PDB with `minAvailable: 1`. This protects voluntary maintenance from removing both replicas simultaneously. It does not make btcdata highly available and does not prevent an involuntary node failure.
### 4.3 Service and ingress
Create a ClusterIP Service mapping port 80 to container port 4040. Create an ingress with:
- ingress class `nginx`;
- host `btcprice.k8s.pmagnus.cloud`;
- path `/`, prefix match;
- backend `btcprice:80`;
- no in-cluster TLS secret, because hostinger Traefik terminates TLS.
Use ingress-nginx timeouts long enough for the two WebSocket endpoints if the controller defaults or shared configuration are shorter than intended. Do not add session affinity unless testing demonstrates a real need.
### 4.4 Flux image automation
Add:
- `ImageRepository` for `harbor.pmagnus.cloud/btcprice/btcprice`;
- registry authentication using Flux's pull-only Harbor secret, not the CI push robot;
- `ImagePolicy` selecting only `main-<14 digit timestamp>` tags numerically;
- the Deployment image marker consumed by the existing `ImageUpdateAutomation`.
Flux should commit the selected tag to `magnus/deploy`. The committed manifest remains the auditable source of truth, and rollback is a Git revert to an earlier immutable tag.
---
## Phase 5 — expose TLS and deploy alongside Compose
1. Add `btcprice.k8s.pmagnus.cloud` as a `tls.domains` entry on hostinger's existing `k8s-ingress` Traefik router. TLS-ALPN-01 cannot issue the wildcard, so this per-app entry is required until DNS-01 is introduced.
2. Edit `traefik-dynamic.yaml` once and allow its file provider to hot-reload. **Do not restart Traefik**; the system record shows that restart can interrupt ACME validation and NetBird traffic.
3. Confirm the certificate is valid externally before treating application errors as Kubernetes errors.
4. Commit the btcprice manifests with a known-good image tag.
5. Confirm Flux reconciliation rather than applying the resources manually.
6. Keep the Compose service on `192.168.50.195:4041` running during this phase. Both deployments can safely use the same stateless btcdata API.
---
## Phase 6 — acceptance tests
### Placement and rollout
- Exactly two Ready btcprice pods exist.
- One is on `srv1893094`; one is on `citadel`.
- Delete each pod in turn and verify it is recreated on the same missing node.
- Push a source change and verify the full unattended chain:
1. Gitea queues the build;
2. `anvil` tests, builds and pushes;
3. Harbor scans the image;
4. Flux selects the new timestamp tag;
5. Flux commits the tag change;
6. kustomize-controller rolls out the Deployment;
7. the URL serves the new version.
Expose the build SHA in application output or a small version endpoint so this can be verified without guessing from pod metadata.
### Functional checks
From outside the cluster and over a sustained test period:
- `https://btcprice.k8s.pmagnus.cloud/` loads with a valid certificate;
- CSS, favicon and external JavaScript assets load;
- `/api/health` and `/api/ready` return 200;
- minute and hourly chart requests return data;
- `/ws/price` and `/ws/strike` stay connected and update;
- ingress logs show successful WebSocket upgrades;
- browser refreshes are served successfully while one pod is deleted;
- a rollout terminates old WebSocket connections cleanly and browser reconnection recovers;
- no image pull, DNS, MTU or cross-node timeout errors appear.
### Failure checks
- Stop one btcprice pod: the site remains usable through the other.
- Temporarily make btcdata unreachable in a controlled window: readiness becomes false without a liveness restart loop, and recovers when btcdata returns.
- Attempt to schedule both replicas on one node: hard anti-affinity must leave the extra replica Pending rather than violate placement.
- Verify a vulnerable/rejected image cannot roll out and the previous ReplicaSet remains available.
**Acceptance criterion:** a push changes the running application with no human deployment step, and the service remains available with either btcprice pod absent.
---
## Phase 7 — cut over and retire the old deployment
After at least 24 hours of successful Kubernetes operation:
1. Record the currently running Compose image and configuration for rollback.
2. Stop only the btcprice Compose project on citadel:
```bash
cd /home/pmp/apps/btcprice
docker compose down
```
Do not stop `btcdata`, `frankfurter` or remove the shared `btc-network`.
3. Confirm port 4041 is no longer needed. If LAN clients still use `http://192.168.50.195:4041`, move them to the HTTPS Kubernetes hostname before closing it.
4. Mark these as legacy/manual-only, but keep them for one rollback window:
- `deploy-citadel.sh`;
- `docker-compose.citadel.yml`;
- `.env.citadel`.
5. After the rollback window, remove the legacy deployment files or move them to a clearly documented archive. Update `README.md` and `CLAUDE.md` so `git push`/Flux is the only documented production deployment path.
6. Update `/Users/pmagnus/pi/system/infrastructure.md` to move btcprice from citadel's Compose list to k3s workloads and record its two-node placement and public URL.
Do not have Compose and Flux both claim the production endpoint. Compose is only a temporary parallel validation and emergency rollback path.
---
## Rollback
### Application release rollback
1. Revert the Flux image-automation commit in `magnus/deploy`, or commit the last known-good immutable tag.
2. Suspend btcprice image automation temporarily if it would immediately select the bad newer tag again.
3. Let Flux reconcile; do not use an unrecorded `kubectl set image` except for immediate incident containment, and commit any emergency change back to Git.
4. Re-enable automation only after the bad image/tag is excluded or replaced.
### Migration rollback
While the Compose rollback window is open:
1. Remove or disable the btcprice ingress in Git and let Flux reconcile.
2. Restart `/home/pmp/apps/btcprice` on citadel with the recorded Compose image/configuration.
3. Restore clients to `http://192.168.50.195:4041` if necessary.
4. Diagnose without deleting Harbor tags or Git history.
### Shared dependency rollback
If the selectorless btcdata Service/EndpointSlice is the fault, revert those manifests. Do not move btcdata itself during the same change window as btcprice; that would combine two independent migrations and obscure failures.
---
## Follow-up work
1. Migrate btcdata and Frankfurter to Kubernetes in a separate plan.
2. Decide how to handle PostgreSQL before migrating btcdata; the current database is native and single-node on citadel.
3. Replace the manual EndpointSlice when btcdata gets a Kubernetes Service.
4. Add monitoring for:
- both btcprice replicas and their node placement;
- Flux reconciliation and image automation failures;
- Harbor scan/retention/GC;
- ingress 5xx and WebSocket disconnect rate;
- btcdata readiness and latency;
- Harbor certificate renewal.
5. Consider Cloudflare DNS-01 for a real `*.k8s.pmagnus.cloud` certificate, removing per-app Traefik certificate entries.
6. Enroll `anvil` in NetBird if desired, then remove the runner `--add-host` and BuildKit host-network workarounds only after a successful test build.
---
## Implementation checklist
- [ ] Flux/CD and `magnus/hello` canary completed
- [ ] Harbor retention, GC, scanning and vulnerability policy configured
- [ ] Shared infrastructure backups verified
- [ ] btcprice branch trigger resolved (`main` versus `master`)
- [ ] stale Dockerfile btcdata port removed/corrected
- [ ] dependency-aware `/api/ready` and tests added
- [ ] local container and WebSocket smoke tests passed
- [ ] Harbor `btcprice` project and least-privilege CI robot created
- [ ] Gitea repository secrets added
- [ ] Gitea Actions build/test/push workflow added
- [ ] scanned immutable image proven pullable on both nodes
- [ ] selectorless btcdata Service and EndpointSlice tested from both nodes
- [ ] two-replica Deployment with required anti-affinity committed
- [ ] Service, ingress, PDB and probes committed
- [ ] Flux ImageRepository/ImagePolicy/automation marker committed
- [ ] Traefik TLS domain added by hot reload, without restart
- [ ] one Ready pod verified on hostinger and one on citadel
- [ ] end-to-end push-to-production test passed
- [ ] functional, node-failure and rollback tests passed
- [ ] 24-hour parallel validation completed
- [ ] old btcprice Compose service stopped without stopping btcdata
- [ ] project and system documentation updated