magnus/btcprice
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document btcprice GitOps deployment
build / build (push) Successful in 2m48s
Details

Browse Source
This commit is contained in:
magnus
2026-09-15 18:35:31 +02:00
parent dd67eb7b23
commit 645c00d8de
1 changed files with 234 additions and 369 deletions
Download Patch File Download Diff File Expand all files Collapse all files
btc-price-deploy.md
+234 -369
View File
@@ -1,251 +1,129 @@
# btcprice CI/CD and Kubernetes deployment plan
# btcprice GitOps migration plan
## Goal
Replace the current workstation-driven deployment:
**Target path**
```text
local docker build -> docker save over SSH -> Docker Compose on citadel
```
with the CI/CD path defined in `/Users/pmagnus/pi/system/cicd-plan.md`:
```text
git push to Gitea
-> Gitea Actions runner `anvil`
-> build and push to Harbor
-> Flux image automation commits a tag update to `magnus/deploy`
git push (magnus/btcprice)
-> Gitea Actions on anvil
-> Harbor image + Trivy scan
-> Flux image policy commits magnus/deploy
-> Flux reconciles k3s
-> ingress-nginx serves btcprice through hostinger's Traefik
-> ingress-nginx
-> hostinger Traefik TLS
-> https://btcprice.k8s.pmagnus.cloud
```
The final Kubernetes Deployment must run **two replicas: exactly one on `srv1893094` (hostinger) and one on `citadel`**. The existing Compose deployment remains available during migration and is removed only after the Kubernetes service is proven.
Production deployment must not use `docker compose`, `kubectl apply` from CI, or a CI-held
kubeconfig. The existing Compose instance on citadel is retained only during validation and as a
short rollback path.
Proposed public URL: **`https://btcprice.k8s.pmagnus.cloud`**.
## Access and secret handling
---
Use `/Users/pmagnus/pi/system/.env` for Gitea, Harbor, host and Cloudflare credentials. The file is
mode `0600` and must remain outside Git.
## Current state and constraints
- Source is already in Gitea at `magnus/btcprice` and currently uses the `master` branch.
- `deploy-citadel.sh` builds locally, copies the image over SSH and starts `docker-compose.citadel.yml` on citadel.
- The current container listens on port `4040`; the Compose deployment publishes it as `192.168.50.195:4041`.
- btcprice is stateless and needs no PVC.
- btcprice depends on btcdata for HTTP and WebSocket upstreams.
- btcdata remains a Compose service on citadel and is published on `0.0.0.0:4102`.
- Both cluster nodes can currently reach btcdata at `http://100.75.8.131:4102`; this was checked from hostinger and citadel and returned HTTP 200 from `/api/health`.
- The k3s cluster already consists of:
- `srv1893094`: server/control plane on hostinger, overlay IP `100.75.252.111`;
- `citadel`: agent, overlay IP `100.75.8.131`.
- Both nodes can pull private Harbor images through the node-level `robot$cluster-pull` credential.
- ingress-nginx is already exposed through hostinger's Traefik at `*.k8s.pmagnus.cloud`.
- Gitea Actions, `anvil`, Harbor builds and scans work, but Flux/CD is not yet installed according to the system plan.
### Important availability boundary
Two btcprice replicas do not make the complete application highly available while btcdata and its PostgreSQL database remain only on citadel. If citadel fails, the hostinger btcprice pod can continue serving static UI and its own health endpoint, but live prices and charts will fail. Migrating or making btcdata highly available is a separate follow-up project.
---
## Target design
```text
Internet
|
v
hostinger Traefik (TLS)
|
v
hostinger ingress-nginx :30080
|
v
ClusterIP Service btcprice:80
|-------------------------------|
v v
btcprice pod btcprice pod
srv1893094 citadel
| |
+---- Service/EndpointSlice ----+
|
v
btcdata at 100.75.8.131:4102
(temporary external dependency)
```bash
set -a
. ../system/.env
set +a
```
Ingress-nginx supports WebSocket upgrades without a btcprice-specific WebSocket annotation. Connections are long-lived and may terminate on either replica; btcprice stores no session state, so no sticky session is required.
Rules:
---
- never echo tokens/passwords or put them in command arguments;
- pass secrets via API headers, stdin, or temporary `0600` files;
- use the Harbor CI robot for image pushes, the existing cluster pull robot for pulls, and Harbor
admin only for project policy changes;
- use the Gitea token for API administration, not from application or workflow code;
- do not copy `../system/.env` into this repository, an image, a pod, or `magnus/deploy`.
## Phase 0 — finish and harden the shared CI/CD road
## Audited starting point (2026-09-15)
Do this before onboarding btcprice. Do not bypass it with a workflow that contains `kubectl` credentials.
Already working:
1. Complete `/Users/pmagnus/pi/system/cicd-plan.md` Phase 3:
- create private Gitea repo `magnus/deploy`;
- bootstrap Flux against `clusters/k3s`;
- install `source-controller`, `kustomize-controller`, `image-reflector-controller` and `image-automation-controller`;
- bring the existing ingress-nginx resources under Flux ownership without deleting the live controller;
- prove the full path using `magnus/hello`.
2. Meet the canary success criterion first: a push to `magnus/hello` must change the response from `https://hello.k8s.pmagnus.cloud` with no manual cluster action.
3. Configure Harbor before btcprice starts producing per-commit tags:
- scan on push;
- prevent pulls of vulnerable images at the agreed severity, initially High;
- retain the last 10 `main-*` builds and all `v*` releases;
- exclude the `buildcache` tag from retention;
- delete untagged artifacts and schedule weekly garbage collection.
4. Prefer creating the planned Docker Hub proxy-cache project before repeated builds. Update the Dockerfile base images to Harbor proxy-cache names only after the cache is verified.
5. Add `100.75.177.123 harbor.pmagnus.cloud` persistently on citadel as described in the system plan, so image pulls do not depend on Technitium running on the same machine.
6. Resolve the shared infrastructure safety items before making deployment automatic:
- back up Harbor, `anvil`, Gitea/deploy and the k3s datastore;
- change `/opt/harbor/harbor.yml` to mode `0600`;
- remove the duplicate ACME issuer for `harbor.pmagnus.cloud`.
- `magnus/btcprice` uses `main`; `.gitea/workflows/build.yml` tests and publishes immutable amd64
images from `anvil`.
- The latest Gitea run for commit `dd67eb7` succeeded.
- Harbor project `btcprice` is private, scans on push, blocks High-or-worse vulnerable images, has a
retention policy and weekly garbage collection. The latest image is scanned successfully:
`main-20260915160353` / `main-20260915160353-dd67eb7`.
- The repository has `HARBOR_ROBOT_USER` and `HARBOR_ROBOT_TOKEN` Actions secrets.
- The image exposes `/api/health`, dependency-aware `/api/ready`, and `/api/version`.
- Flux and all image automation controllers are healthy. The `hello` canary proves unattended image
update, reconciliation, two-node scheduling, ingress, and TLS.
- k3s has two Ready amd64 nodes: `srv1893094` and `citadel`.
- Both nodes already have Harbor pull credentials.
- ingress-nginx is managed by Flux and is reached through hostinger Traefik on NodePort 30080.
- btcdata still runs under Compose on citadel and answers on `100.75.8.131:4102`.
- The old btcprice Compose container remains available on citadel port 4041.
**Gate:** do not proceed until Flux reconciliation, image reflection/automation, private image pulls on both nodes and the canary ingress all work.
Current gaps:
---
- `magnus/deploy` has no btcprice resources or image policy.
- `btcprice.k8s.pmagnus.cloud` currently reaches Traefik but receives its default certificate and an
nginx 404; the TLS domain and Kubernetes Ingress do not exist yet.
- Local `../deploy` can lag Flux's automated commits. Always run `git pull --ff-only` before editing.
- Harbor's release retention rule currently keeps `v*`, while the workflow strips the leading `v`
(`v1.2.3` becomes `1.2.3`). Correct that mismatch before relying on release retention.
- The Gitea btcprice repository is currently public. Confirm that this is intentional; it does not
expose the Actions secrets, but it does expose source and workflow definitions.
## Phase 1 — make btcprice deployment-ready
## Phase 1 — close pre-deployment policy gaps
### 1.1 Keep runtime configuration external
1. In Harbor, change the release retention rule to match the tags the workflow actually emits
(plain semantic versions and their SHA-qualified companion tags), or change the workflow to retain
the `v` prefix. Prefer changing retention so existing image naming remains stable.
2. Confirm `buildcache` remains explicitly retained and excluded from application deployment policy.
3. Confirm the exact latest digest is pullable on both k3s nodes after its Trivy scan passes.
4. Run `make test` locally and confirm the published image reports `dd67eb7` from `/api/version`.
5. Decide whether `magnus/btcprice` should remain public.
The Kubernetes manifest, not the image, must set:
**Gate:** no Kubernetes manifests are merged until tests pass, the latest immutable image is approved
by Harbor, and both nodes can pull it.
## Phase 2 — represent the external btcdata dependency in Kubernetes
Create the following in `../deploy/apps/btcprice/`:
- `namespace.yaml`
- `btcdata-service.yaml`
- `btcdata-endpointslice.yaml`
Use namespace `btcprice`. Define a selectorless ClusterIP Service named `btcdata` on port `4102` and
an EndpointSlice with:
- label `kubernetes.io/service-name: btcdata`;
- address type `IPv4`;
- endpoint `100.75.8.131`;
- TCP port `4102`.
The application then uses only:
```text
BTCDATA_URL=http://btcdata:4102
```
Remove or correct the stale Dockerfile default `http://btcdata:4101`. A production image should not silently point to the wrong port. Retain `http://localhost:4100` as the application's local-development default if desired.
Do not embed citadel's IP in the Deployment. This abstraction lets btcdata move into Kubernetes later
without changing btcprice.
### 1.2 Add dependency-aware readiness
Test from disposable pods pinned once to `srv1893094` and once to `citadel`:
`GET /api/health` currently proves only that FastAPI is running. Keep it as the liveness endpoint and add a readiness endpoint, for example `GET /api/ready`, which performs a short, bounded request to `${BTCDATA_URL}/api/health`:
- `http://btcdata:4102/api/health` returns 200;
- `/api/kraken-minute` and `/api/kraken-hour` return JSON;
- WebSocket connections to `/api/price/ws` and `/api/strike/ws` receive data.
- return 200 only when btcdata answers successfully;
- use a 2–3 second timeout;
- return 503 without exposing internal exception details;
- do not use readiness for liveness, otherwise a btcdata outage would restart healthy btcprice pods continuously.
**Gate:** all HTTP and WebSocket checks pass from both nodes.
Add tests for the ready and unavailable cases. Add `pytest` and FastAPI's test client dependencies to a development/test requirements file rather than the production image if practical.
## Phase 3 — add the btcprice workload to `magnus/deploy`
### 1.3 Validate container behavior
First synchronize the GitOps checkout:
Before enabling CI:
1. Build the image locally.
2. Run it with `BTCDATA_URL=http://100.75.8.131:4102`.
3. Verify:
- `/api/health` returns 200;
- `/api/ready` returns 200;
- both chart proxy endpoints return JSON;
- `/ws/price` and `/ws/strike` receive updates;
- the process exits cleanly on SIGTERM within the planned pod termination grace period.
4. Optionally add a non-root runtime user. If done, confirm static files remain readable and Uvicorn can bind to port 4040.
### 1.4 Branch decision
The shared workflow template watches `main`, while this repo currently uses `master`. Choose one explicitly:
- preferred: rename the default branch to `main` in Git and Gitea, then update local tracking; or
- keep `master` and set the workflow trigger to `branches: [master]`.
Do not copy the template unchanged or pushes will not trigger builds. The remaining examples assume `main`.
---
## Phase 2 — Harbor and Gitea Actions build pipeline
### 2.1 Harbor repository and credentials
Create a private Harbor project `btcprice` and use image:
```text
harbor.pmagnus.cloud/btcprice/btcprice
```bash
git -C ../deploy pull --ff-only
git -C ../deploy switch main
```
Create a project-scoped robot such as `robot$btcprice+ci` with only the permissions needed to push and pull artifacts in this project. Verify that it cannot access unrelated projects or perform administrative deletion.
Store its one-time secret in the **btcprice Gitea repository** as:
- `HARBOR_ROBOT_USER`
- `HARBOR_ROBOT_TOKEN`
Do not use Harbor admin credentials and do not place the token in the runner environment.
The existing system-level `robot$cluster-pull` already covers future projects through `namespace: "*"`; verify a private pull from the new project on both nodes before relying on it.
### 2.2 Add `.gitea/workflows/build.yml`
Base it on the proven `magnus/hello` workflow, with these btcprice-specific values:
```yaml
env:
REGISTRY: harbor.pmagnus.cloud
IMAGE: harbor.pmagnus.cloud/btcprice/btcprice
```
Required behavior:
1. Trigger on pushes to `main` and tags matching `v*`.
2. Check out source.
3. Run automated tests before publishing.
4. Produce immutable tags:
- branch: `main-<UTC YYYYMMDDHHMMSS>` and `main-<timestamp>-<short SHA>`;
- release: version without the leading `v`, plus a SHA-qualified tag.
5. Log in with the project robot.
6. Build for `linux/amd64`; both current cluster nodes are amd64.
7. Push through BuildKit with registry cache:
- `cache-from: type=registry,ref=...:buildcache`;
- `cache-to: type=registry,ref=...:buildcache,mode=max`.
8. Preserve the working `anvil` networking workaround from the canary pipeline: use `driver-opts: network=host` in `docker/setup-buildx-action`, because `anvil` is not a NetBird peer. Its runner config must retain the Harbor `--add-host` option.
9. Do not provide a kubeconfig or deploy from the workflow.
The Flux policy must match only the plain timestamp tag, not the second SHA-qualified tag. Use a pattern equivalent to:
```text
^main-(?P<ts>[0-9]{14})$
```
and numerical ordering by `ts`.
### 2.3 Prove the build half independently
Push a harmless commit and verify:
- the Gitea Actions run completes on `anvil`;
- Harbor contains both immutable tags and `buildcache`;
- Trivy scan completes;
- the manifest architecture is amd64;
- each k3s node can independently pull the exact digest;
- no `latest` tag is required for deployment.
Stop and fix the build path before creating application manifests if any check fails.
---
## Phase 3 — describe btcdata temporarily inside Kubernetes
Do not put the overlay IP directly in the btcprice Deployment. Add an abstraction in `magnus/deploy`, under the btcprice namespace:
- a selectorless Service named `btcdata`, port `4102`;
- an EndpointSlice for that Service with endpoint `100.75.8.131`, port `4102`.
This gives btcprice the stable URL `http://btcdata:4102`. When btcdata is later migrated into Kubernetes, replace the EndpointSlice with normal pod selectors without changing btcprice.
Before committing it, verify from a temporary pod pinned to **each node** that:
```text
http://btcdata:4102/api/health
```
returns 200 and that a WebSocket connection can be established.
Security follow-up: btcdata currently publishes `4102` on all interfaces. Limit access to the NetBird/cluster path with a tested firewall rule or a safe Docker binding. Do not change its binding in a way that races NetBird startup or breaks the existing Compose `btc-network` connection.
---
## Phase 4 — add btcprice to the GitOps repository
Create this structure in `magnus/deploy`:
Add this structure:
```text
apps/btcprice/
@@ -256,211 +134,198 @@ apps/btcprice/
service.yaml
ingress.yaml
pdb.yaml
image-repository.yaml
image-policy.yaml
kustomization.yaml
infrastructure/image-automation/
btcprice-image.yaml
```
Reference it from the cluster's applications Kustomization.
Also update:
### 4.1 Deployment
- `apps/kustomization.yaml` to include `btcprice`;
- `infrastructure/image-automation/kustomization.yaml` to include `btcprice-image.yaml`.
### Deployment specification
Use:
- `replicas: 2`;
- image `harbor.pmagnus.cloud/btcprice/btcprice:<initial-known-good-tag>`;
- Flux image-policy marker on the image field;
- container port `4040`;
- image `harbor.pmagnus.cloud/btcprice/btcprice:<known-good-main-tag>`;
- Flux marker `# {"$imagepolicy": "flux-system:btcprice"}` on the image;
- two replicas and container port `4040`;
- `BTCDATA_URL=http://btcdata:4102`;
- conservative CPU/memory requests and limits based on a measured Compose container, then tune from metrics;
- `terminationGracePeriodSeconds` sufficient to close WebSockets cleanly;
- rolling strategy `maxUnavailable: 1`, `maxSurge: 0`.
- `terminationGracePeriodSeconds: 30`;
- rolling update with `maxUnavailable: 1` and `maxSurge: 0`;
- initial resources of approximately `50m` CPU / `64Mi` memory requested and `250m` CPU / `256Mi`
memory limited, then tune from observed metrics.
Guarantee one replica per cluster node with both:
Guarantee one replica on each node with:
1. node affinity restricting eligible hosts to `srv1893094` and `citadel`;
2. **required** pod anti-affinity on `kubernetes.io/hostname` for pods with `app.kubernetes.io/name=btcprice`.
- required node affinity allowing only hostnames `srv1893094` and `citadel`;
- required pod anti-affinity on `kubernetes.io/hostname`, matching
`app.kubernetes.io/name: btcprice`.
Hard anti-affinity is intentional: with exactly two eligible nodes and two replicas it guarantees one pod on each. `maxSurge: 0` avoids a rollout deadlock, because a third surge pod cannot satisfy hard anti-affinity.
Hard anti-affinity plus two replicas and two eligible nodes gives one pod per node. `maxSurge: 0`
prevents rollout deadlock because a third pod cannot satisfy that anti-affinity.
Add probes:
Use probes:
- startup: `/api/health` with enough allowance for startup/image extraction;
- liveness: `/api/health`;
- readiness: `/api/ready`.
- startup: `GET /api/health`;
- liveness: `GET /api/health`;
- readiness: `GET /api/ready`, with Kubernetes timeout longer than the application's three-second
upstream timeout.
Add standard labels and make selectors immutable and consistent.
Never use dependency-aware readiness as liveness; a btcdata outage must not restart healthy
btcprice processes.
### 4.2 Pod disruption budget
### Service, disruption budget, and ingress
Create a PDB with `minAvailable: 1`. This protects voluntary maintenance from removing both replicas simultaneously. It does not make btcdata highly available and does not prevent an involuntary node failure.
- ClusterIP Service: port `80` to named container port `4040`.
- PDB: `minAvailable: 1`.
- Ingress class: `nginx`.
- Host: `btcprice.k8s.pmagnus.cloud`.
- Path: `/`, `Prefix`, backend `btcprice:80`.
- No Kubernetes TLS secret: TLS terminates at hostinger Traefik.
- Add ingress-nginx proxy read/send timeouts suitable for long-lived WebSockets (for example 3600
seconds). WebSocket upgrade support itself is automatic.
### 4.3 Service and ingress
## Phase 4 — add Flux image automation
Create a ClusterIP Service mapping port 80 to container port 4040. Create an ingress with:
Follow the existing `hello` convention in `../deploy/infrastructure/image-automation/`:
- ingress class `nginx`;
- host `btcprice.k8s.pmagnus.cloud`;
- path `/`, prefix match;
- backend `btcprice:80`;
- no in-cluster TLS secret, because hostinger Traefik terminates TLS.
- `ImageRepository` named `btcprice` in `flux-system`;
- image `harbor.pmagnus.cloud/btcprice/btcprice`;
- existing pull-only secret `harbor-pull`;
- one-minute scan interval;
- `ImagePolicy` matching only plain branch tags:
Use ingress-nginx timeouts long enough for the two WebSocket endpoints if the controller defaults or shared configuration are shorter than intended. Do not add session affinity unless testing demonstrates a real need.
```yaml
filterTags:
pattern: '^main-(?P<ts>[0-9]{14})$'
extract: '$ts'
policy:
numerical:
order: asc
```
### 4.4 Flux image automation
This intentionally excludes SHA-qualified tags, releases, and `buildcache`. The existing
`ImageUpdateAutomation/apps` object will update files under `./apps`, commit to `magnus/deploy`, and
cause Flux to reconcile. CI receives no GitOps write key or kubeconfig.
Add:
Validate manifests before pushing:
- `ImageRepository` for `harbor.pmagnus.cloud/btcprice/btcprice`;
- registry authentication using Flux's pull-only Harbor secret, not the CI push robot;
- `ImagePolicy` selecting only `main-<14 digit timestamp>` tags numerically;
- the Deployment image marker consumed by the existing `ImageUpdateAutomation`.
```bash
kubectl kustomize ../deploy/apps/btcprice >/dev/null
kubectl kustomize ../deploy/apps >/dev/null
kubectl kustomize ../deploy/infrastructure/image-automation >/dev/null
```
Flux should commit the selected tag to `magnus/deploy`. The committed manifest remains the auditable source of truth, and rollback is a Git revert to an earlier immutable tag.
Commit and push only to `magnus/deploy`; do not manually apply the resources. Watch Flux and verify
that all Kustomizations remain Ready.
---
## Phase 5 — enable ingress TLS
## Phase 5 — expose TLS and deploy alongside Compose
On hostinger, edit the file-provider configuration mounted at:
1. Add `btcprice.k8s.pmagnus.cloud` as a `tls.domains` entry on hostinger's existing `k8s-ingress` Traefik router. TLS-ALPN-01 cannot issue the wildcard, so this per-app entry is required until DNS-01 is introduced.
2. Edit `traefik-dynamic.yaml` once and allow its file provider to hot-reload. **Do not restart Traefik**; the system record shows that restart can interrupt ACME validation and NetBird traffic.
3. Confirm the certificate is valid externally before treating application errors as Kubernetes errors.
4. Commit the btcprice manifests with a known-good image tag.
5. Confirm Flux reconciliation rather than applying the resources manually.
6. Keep the Compose service on `192.168.50.195:4041` running during this phase. Both deployments can safely use the same stateless btcdata API.
```text
/home/pmp/apps/netbird/traefik-dynamic.yaml
```
---
Add this entry under `http.routers.k8s-ingress.tls.domains`:
## Phase 6 — acceptance tests
```yaml
- main: "btcprice.k8s.pmagnus.cloud"
```
### Placement and rollout
Use `SUDO_HOSTINGER` from `../system/.env` only if privilege escalation is required. Back up the file
first and preserve permissions. Do **not** restart Traefik: its file provider hot-reloads, and a
restart can interrupt ACME and NetBird traffic.
- Exactly two Ready btcprice pods exist.
- One is on `srv1893094`; one is on `citadel`.
- Delete each pod in turn and verify it is recreated on the same missing node.
- Push a source change and verify the full unattended chain:
1. Gitea queues the build;
2. `anvil` tests, builds and pushes;
3. Harbor scans the image;
4. Flux selects the new timestamp tag;
5. Flux commits the tag change;
6. kustomize-controller rolls out the Deployment;
7. the URL serves the new version.
Wait for issuance and verify externally:
Expose the build SHA in application output or a small version endpoint so this can be verified without guessing from pod metadata.
```bash
curl --fail https://btcprice.k8s.pmagnus.cloud/api/health
curl --fail https://btcprice.k8s.pmagnus.cloud/api/ready
```
### Functional checks
The certificate must be publicly trusted and named for `btcprice.k8s.pmagnus.cloud`; do not accept the
current self-signed `TRAEFIK DEFAULT CERT`.
From outside the cluster and over a sustained test period:
## Phase 6 — acceptance and unattended deployment test
- `https://btcprice.k8s.pmagnus.cloud/` loads with a valid certificate;
- CSS, favicon and external JavaScript assets load;
- `/api/health` and `/api/ready` return 200;
- minute and hourly chart requests return data;
- `/ws/price` and `/ws/strike` stay connected and update;
- ingress logs show successful WebSocket upgrades;
- browser refreshes are served successfully while one pod is deleted;
- a rollout terminates old WebSocket connections cleanly and browser reconnection recovers;
- no image pull, DNS, MTU or cross-node timeout errors appear.
### Initial deployment
### Failure checks
Confirm:
- Stop one btcprice pod: the site remains usable through the other.
- Temporarily make btcdata unreachable in a controlled window: readiness becomes false without a liveness restart loop, and recovers when btcdata returns.
- Attempt to schedule both replicas on one node: hard anti-affinity must leave the extra replica Pending rather than violate placement.
- Verify a vulnerable/rejected image cannot roll out and the previous ReplicaSet remains available.
- both Flux application and infrastructure Kustomizations are Ready;
- `ImageRepository`, `ImagePolicy`, and `ImageUpdateAutomation` are Ready;
- exactly two btcprice pods are Ready, one on each node;
- Service endpoints contain both pods;
- `/`, CSS, favicon, chart APIs, health, readiness, and version endpoints work externally;
- `/ws/price` and `/ws/strike` remain connected and update through ingress;
- the running `/api/version` equals the expected image's embedded short SHA.
**Acceptance criterion:** a push changes the running application with no human deployment step, and the service remains available with either btcprice pod absent.
### Full pipeline test
---
Push a harmless visible change to `magnus/btcprice` and observe, without manual deployment actions:
## Phase 7 — cut over and retire the old deployment
1. Gitea queues the workflow on `anvil`.
2. Tests pass.
3. Harbor receives the timestamp and SHA-qualified tags.
4. Trivy approves the artifact.
5. Flux image reflection selects the new plain timestamp tag.
6. Flux commits the image change to `magnus/deploy`.
7. kustomize-controller rolls out the Deployment.
8. `/api/version` changes to the pushed source SHA.
After at least 24 hours of successful Kubernetes operation:
### Failure tests
1. Record the currently running Compose image and configuration for rollback.
2. Stop only the btcprice Compose project on citadel:
- Delete each pod in turn; one endpoint remains available and the missing node gets a replacement.
- During a rollout, verify browser WebSockets reconnect after old pods terminate.
- Temporarily make btcdata unreachable in a controlled window; readiness becomes false, liveness
remains true, and no restart loop occurs.
- Verify a Harbor-rejected image cannot become a healthy rollout.
- Revert a GitOps image commit and confirm Flux restores the prior immutable image.
```bash
cd /home/pmp/apps/btcprice
docker compose down
```
**Acceptance criterion:** a normal source push changes production with no SSH, Compose, or manual
`kubectl` deployment, while Git records the deployed image.
Do not stop `btcdata`, `frankfurter` or remove the shared `btc-network`.
3. Confirm port 4041 is no longer needed. If LAN clients still use `http://192.168.50.195:4041`, move them to the HTTPS Kubernetes hostname before closing it.
4. Mark these as legacy/manual-only, but keep them for one rollback window:
- `deploy-citadel.sh`;
- `docker-compose.citadel.yml`;
- `.env.citadel`.
5. After the rollback window, remove the legacy deployment files or move them to a clearly documented archive. Update `README.md` and `CLAUDE.md` so `git push`/Flux is the only documented production deployment path.
6. Update `/Users/pmagnus/pi/system/infrastructure.md` to move btcprice from citadel's Compose list to k3s workloads and record its two-node placement and public URL.
## Phase 7 — cut over from Compose
Do not have Compose and Flux both claim the production endpoint. Compose is only a temporary parallel validation and emergency rollback path.
Keep `http://192.168.50.195:4041` running in parallel for at least 24 hours after all acceptance tests
pass. It must not own the new Kubernetes hostname.
---
Then on citadel:
1. Record the legacy image ID and effective environment for rollback without recording secrets.
2. Stop only `/home/pmp/apps/btcprice`; do not stop btcdata, Frankfurter, PostgreSQL, or remove
`btc-network`.
3. Verify no clients still use port 4041.
4. Keep `deploy-citadel.sh`, `docker-compose.citadel.yml`, and `.env.citadel` for one defined rollback
window, clearly marked legacy/manual-only.
5. After that window, remove the legacy deployment files and update `README.md`, `CLAUDE.md`, and
`../system/infrastructure.md` so GitOps is the only production path.
## Rollback
### Application release rollback
Application rollback:
1. Revert the Flux image-automation commit in `magnus/deploy`, or commit the last known-good immutable tag.
2. Suspend btcprice image automation temporarily if it would immediately select the bad newer tag again.
3. Let Flux reconcile; do not use an unrecorded `kubectl set image` except for immediate incident containment, and commit any emergency change back to Git.
4. Re-enable automation only after the bad image/tag is excluded or replaced.
1. Suspend the btcprice `ImagePolicy` or image automation if it would immediately reselect the bad
tag.
2. Revert the automated image commit in `magnus/deploy`, or commit the previous known-good immutable
tag.
3. Let Flux reconcile and verify readiness/version.
4. Re-enable automation only after the bad tag is no longer selectable.
### Migration rollback
Migration rollback while Compose is retained:
While the Compose rollback window is open:
1. Remove or suspend the btcprice Ingress through Git and let Flux reconcile.
2. Restart only the recorded btcprice Compose service on citadel.
3. Restore clients temporarily to port 4041.
4. Diagnose without deleting Harbor artifacts or rewriting Git history.
1. Remove or disable the btcprice ingress in Git and let Flux reconcile.
2. Restart `/home/pmp/apps/btcprice` on citadel with the recorded Compose image/configuration.
3. Restore clients to `http://192.168.50.195:4041` if necessary.
4. Diagnose without deleting Harbor tags or Git history.
## Known limitation and follow-up
### Shared dependency rollback
If the selectorless btcdata Service/EndpointSlice is the fault, revert those manifests. Do not move btcdata itself during the same change window as btcprice; that would combine two independent migrations and obscure failures.
---
## Follow-up work
1. Migrate btcdata and Frankfurter to Kubernetes in a separate plan.
2. Decide how to handle PostgreSQL before migrating btcdata; the current database is native and single-node on citadel.
3. Replace the manual EndpointSlice when btcdata gets a Kubernetes Service.
4. Add monitoring for:
- both btcprice replicas and their node placement;
- Flux reconciliation and image automation failures;
- Harbor scan/retention/GC;
- ingress 5xx and WebSocket disconnect rate;
- btcdata readiness and latency;
- Harbor certificate renewal.
5. Consider Cloudflare DNS-01 for a real `*.k8s.pmagnus.cloud` certificate, removing per-app Traefik certificate entries.
6. Enroll `anvil` in NetBird if desired, then remove the runner `--add-host` and BuildKit host-network workarounds only after a successful test build.
---
## Implementation checklist
- [x] Flux/CD and `magnus/hello` canary completed
- [x] Harbor retention, GC, scanning and vulnerability policy configured
- [x] Shared infrastructure backups verified
- [x] btcprice branch trigger resolved (`main` versus `master`)
- [x] stale Dockerfile btcdata port removed/corrected
- [x] dependency-aware `/api/ready` and tests added
- [x] local container and WebSocket smoke tests passed
- [ ] Harbor `btcprice` project and least-privilege CI robot created
- [ ] Gitea repository secrets added
- [ ] Gitea Actions build/test/push workflow added
- [ ] scanned immutable image proven pullable on both nodes
- [ ] selectorless btcdata Service and EndpointSlice tested from both nodes
- [ ] two-replica Deployment with required anti-affinity committed
- [ ] Service, ingress, PDB and probes committed
- [ ] Flux ImageRepository/ImagePolicy/automation marker committed
- [ ] Traefik TLS domain added by hot reload, without restart
- [ ] one Ready pod verified on hostinger and one on citadel
- [ ] end-to-end push-to-production test passed
- [ ] functional, node-failure and rollback tests passed
- [ ] 24-hour parallel validation completed
- [ ] old btcprice Compose service stopped without stopping btcdata
- [ ] project and system documentation updated
Two frontend replicas do not make the full system highly available: btcdata and its database remain
single-node dependencies on citadel. If citadel fails, the hostinger btcprice pod may serve static
content but readiness and live data fail. Migrate btcdata and decide PostgreSQL storage/availability
in a separate change; then replace the selectorless Service and EndpointSlice with a normal
Kubernetes Service.