Observed 2026-09-19: one Kubernetes btcdata pod on citadel, both eligible nodes Ready, but at 14:03 UTC its WebSocket returned a price timestamped 08:36:42 UTC. HTTP 101 and existing readiness did not detect the stalled upstream. Harbor refuses connections; Flux apps reconciliation is blocked. No live workload mutation was performed.
Changes: supervised Binance reconnects after errors/closures/silence; 60-second in-process freshness readiness; PostgreSQL session advisory lock with ALL application queries on the same lock-owning connection, preventing reconnect-through-pool split-brain writes; bounded DB network waits; secret-free PR tests.
Verified locally: 9 Clojure tests / 25 assertions pass including disposable PostgreSQL contention, lost-session and pooled-lock release tests; same suite passes in Java 21 containers via scripts/test-fencing.sh; make uberjar succeeds; production JAR starts against a disposable DB, migrates, passes readiness with external pollers disabled, and exits on SIGTERM in 0.3 seconds. Existing CRLF source formatting preserved. No credentials staged.
Companion magnus/deploy branch fix/btcdata-websocket-failover adds a SUSPENDED watchdog and rollout runbook. Do not enable relocation until this image is built, scanned, pulled on both nodes and verified. All legacy/unfenced writers must remain stopped. DB remains a citadel single point of failure.
Owner review/merge required per system/PR-DEPLOYMENT.md. No production failover acceptance tests or image rollout completed. Rollback: suspend watchdog first, ensure one writer, revert approved image via Flux Recreate; remove writer-fenced annotation if reverting fencing. Observe the PR status context before making it required in branch protection.
Observed 2026-09-19: one Kubernetes btcdata pod on citadel, both eligible nodes Ready, but at 14:03 UTC its WebSocket returned a price timestamped 08:36:42 UTC. HTTP 101 and existing readiness did not detect the stalled upstream. Harbor refuses connections; Flux apps reconciliation is blocked. No live workload mutation was performed.
Changes: supervised Binance reconnects after errors/closures/silence; 60-second in-process freshness readiness; PostgreSQL session advisory lock with ALL application queries on the same lock-owning connection, preventing reconnect-through-pool split-brain writes; bounded DB network waits; secret-free PR tests.
Verified locally: 9 Clojure tests / 25 assertions pass including disposable PostgreSQL contention, lost-session and pooled-lock release tests; same suite passes in Java 21 containers via scripts/test-fencing.sh; make uberjar succeeds; production JAR starts against a disposable DB, migrates, passes readiness with external pollers disabled, and exits on SIGTERM in 0.3 seconds. Existing CRLF source formatting preserved. No credentials staged.
Companion magnus/deploy branch fix/btcdata-websocket-failover adds a SUSPENDED watchdog and rollout runbook. Do not enable relocation until this image is built, scanned, pulled on both nodes and verified. All legacy/unfenced writers must remain stopped. DB remains a citadel single point of failure.
Owner review/merge required per system/PR-DEPLOYMENT.md. No production failover acceptance tests or image rollout completed. Rollback: suspend watchdog first, ensure one writer, revert approved image via Flux Recreate; remove writer-fenced annotation if reverting fencing. Observe the PR status context before making it required in branch protection.
Current head c0843af passes pr / test (pull_request). Inspected CI logs: 9 tests, 25 assertions, zero failures/errors (not an empty-workspace success). Fixed nested-Docker workspace copying in both PR and image-build tests; checkout credentials are not persisted in PR jobs. Local production JAR startup/migrations/readiness and SIGTERM also passed. Companion: magnus/deploy#6 . Live deployment remains pending owner review and Harbor repair; no production mutation was performed.
Current head c0843af passes pr / test (pull_request). Inspected CI logs: 9 tests, 25 assertions, zero failures/errors (not an empty-workspace success). Fixed nested-Docker workspace copying in both PR and image-build tests; checkout credentials are not persisted in PR jobs. Local production JAR startup/migrations/readiness and SIGTERM also passed. Companion: https://git.pmagnus.cloud/magnus/deploy/pulls/6 . Live deployment remains pending owner review and Harbor repair; no production mutation was performed.
Live follow-up, 2026-09-19 through 14:40 UTC: Harbor restored by starting its failed containers; all image scans and Flux Kustomizations now Ready. Controlled one-replica Recreate relocation to hostinger failed PostgreSQL authorization, not routing: JDBC already targets NetBird 100.75.8.131:5432/btcprod; PostgreSQL rejects client 100.75.252.111 (postgres/btcprod, no HBA rule). Rolled back to citadel, one Ready pod, price WebSocket verified delivering four advancing timestamps. Runtime selector temporarily pins citadel and survives Flux reconciliation. Citadel sudo credential was rejected once; no HBA changes made, temporary credentials removed. Correct administrative access and a narrowly scoped authenticated HBA rule are needed before retrying. Automatic failover remains disabled; PRs remain pending owner review.
Live follow-up, 2026-09-19 through 14:40 UTC: Harbor restored by starting its failed containers; all image scans and Flux Kustomizations now Ready. Controlled one-replica Recreate relocation to hostinger failed PostgreSQL authorization, not routing: JDBC already targets NetBird 100.75.8.131:5432/btcprod; PostgreSQL rejects client 100.75.252.111 (postgres/btcprod, no HBA rule). Rolled back to citadel, one Ready pod, price WebSocket verified delivering four advancing timestamps. Runtime selector temporarily pins citadel and survives Flux reconciliation. Citadel sudo credential was rejected once; no HBA changes made, temporary credentials removed. Correct administrative access and a narrowly scoped authenticated HBA rule are needed before retrying. Automatic failover remains disabled; PRs remain pending owner review.
magnus
merged commit bd25fec56b into master2026-09-19 18:09:20 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Observed 2026-09-19: one Kubernetes btcdata pod on citadel, both eligible nodes Ready, but at 14:03 UTC its WebSocket returned a price timestamped 08:36:42 UTC. HTTP 101 and existing readiness did not detect the stalled upstream. Harbor refuses connections; Flux apps reconciliation is blocked. No live workload mutation was performed.
Changes: supervised Binance reconnects after errors/closures/silence; 60-second in-process freshness readiness; PostgreSQL session advisory lock with ALL application queries on the same lock-owning connection, preventing reconnect-through-pool split-brain writes; bounded DB network waits; secret-free PR tests.
Verified locally: 9 Clojure tests / 25 assertions pass including disposable PostgreSQL contention, lost-session and pooled-lock release tests; same suite passes in Java 21 containers via scripts/test-fencing.sh; make uberjar succeeds; production JAR starts against a disposable DB, migrates, passes readiness with external pollers disabled, and exits on SIGTERM in 0.3 seconds. Existing CRLF source formatting preserved. No credentials staged.
Companion magnus/deploy branch fix/btcdata-websocket-failover adds a SUSPENDED watchdog and rollout runbook. Do not enable relocation until this image is built, scanned, pulled on both nodes and verified. All legacy/unfenced writers must remain stopped. DB remains a citadel single point of failure.
Owner review/merge required per system/PR-DEPLOYMENT.md. No production failover acceptance tests or image rollout completed. Rollback: suspend watchdog first, ensure one writer, revert approved image via Flux Recreate; remove writer-fenced annotation if reverting fencing. Observe the PR status context before making it required in branch protection.
Current head
c0843afpasses pr / test (pull_request). Inspected CI logs: 9 tests, 25 assertions, zero failures/errors (not an empty-workspace success). Fixed nested-Docker workspace copying in both PR and image-build tests; checkout credentials are not persisted in PR jobs. Local production JAR startup/migrations/readiness and SIGTERM also passed. Companion: magnus/deploy#6 . Live deployment remains pending owner review and Harbor repair; no production mutation was performed.Live follow-up, 2026-09-19 through 14:40 UTC: Harbor restored by starting its failed containers; all image scans and Flux Kustomizations now Ready. Controlled one-replica Recreate relocation to hostinger failed PostgreSQL authorization, not routing: JDBC already targets NetBird 100.75.8.131:5432/btcprod; PostgreSQL rejects client 100.75.252.111 (postgres/btcprod, no HBA rule). Rolled back to citadel, one Ready pod, price WebSocket verified delivering four advancing timestamps. Runtime selector temporarily pins citadel and survives Flux reconciliation. Citadel sudo credential was rejected once; no HBA changes made, temporary credentials removed. Correct administrative access and a narrowly scoped authenticated HBA rule are needed before retrying. Automatic failover remains disabled; PRs remain pending owner review.