magnus/btcdata
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fence market-data writes and recover stalled Binance streams
pr / test (pull_request) Successful in 1m22s
Details

Browse Source
This commit is contained in:
magnus
2026-09-19 16:16:48 +02:00
parent c549da40e1
commit 904f3fffe6
12 changed files with 336 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
btcdata-deploy.md
+27
View File
@@ -1,5 +1,32 @@
# btcdata Kubernetes deployment plan
## 2026-09-19 status and proposed recovery
Kubernetes deployment already exists: one `Recreate` replica on citadel, eligible
on citadel and srv1893094. Both nodes were Ready. At 14:03 UTC the live price
WebSocket upgraded successfully but returned data last recorded at 08:36:42 UTC.
Harbor was refusing connections and Flux apps reconciliation was dependency-blocked.
No production restart or relocation was performed during this investigation.
The `fix/websocket-failover` application change proposes bounded Binance reconnects,
60-second in-memory freshness readiness, and PostgreSQL single-writer fencing.
Every application query uses the session holding advisory lock `728194106`; loss
of that session cannot cause writes through another pooled connection. Competing
instances fail startup rather than becoming simultaneous writers. This requires
all writers to run the fenced version; legacy deployments must stay stopped.
Companion deployment changes in `magnus/deploy` add a **suspended** watchdog:
three failed minute checks after startup grace request relocation to the other
healthy node, with a ten-minute cooldown. See `apps/btcdata/FAILOVER.md` in that
repository for review gates, acceptance tests, limitations, and rollback. This is
proposed code, not a claim that automatic failover is enabled in production.
Run `make test` for unit tests. Run `bash scripts/test-fencing.sh` for unit tests
plus PostgreSQL lock-contention, session-loss, and pooled-lock-release tests in
isolated disposable containers. No production credentials are required.
The remaining sections below are the original migration plan, not current status.
## Goal and interpretation
Deploy `btcdata` through the existing Flux-managed two-node k3s cluster (`srv1893094` and `citadel`), with **exactly one btcdata pod running at a time**. The pod may run on either node and Kubernetes may recreate it on the other node after failure.