diff --git a/PLAN.md b/PLAN.md index 41fa833..fd62d59 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,8 +1,8 @@ # btcdata-archive implementation handoff -Status: **IMPLEMENTED LOCALLY — restricted RustFS IAM identity, versioned archive bucket, PostgreSQL ledger and restricted runtime role provisioned on explicit requests; source index preflight, deployment and deletion have not been performed.** -Implementation artifacts and validation instructions are in `README.md`. The disposable -PostgreSQL/RustFS suite is prepared but requires a working Docker environment before rollout. +Status: **IMPLEMENTED AND DEPLOYED SUSPENDED — restricted identities, versioned bucket, ledger and indexes provisioned on explicit requests. The operator completed the initial catch-up. CI passed 65 tests; a verify-only Kubernetes Job completed on Citadel. Recurring pruning remains disabled pending separate approval.** +Implementation artifacts, deployment state and validation results are in `README.md`. +The original requirements below remain the retention/archive contract. Prepared: 2026-10-03. Target repository: `/Users/pmagnus/pi/btcdata-archive`. diff --git a/README.md b/README.md index 0633daa..def7142 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ Standalone Python archiver for the three fixed `btcprod` tables: `public.binance_price`, `public.strike_price`, and `public.kraken_minute`. **A restricted RustFS IAM identity and versioned archive bucket, progress ledger and restricted PostgreSQL role have -been provisioned; source index preflight and deployment are still pending.** -An operator run verified the first weekly archive, then hit a ledger transition bug -before deletion. That bug is now fixed; deployment is being prepared suspended. +been provisioned. The Kubernetes CronJob is deployed through GitOps, suspended.** +The initial ledger transition bug is fixed; the operator completed 28 backlog weeks. +Rollout validation ran verify-only and did not delete any source rows. The repository is hosted at https://git.pmagnus.cloud/magnus/btcdata-archive; Gitea Actions and project-scoped Harbor CI/pull credentials are configured. @@ -30,10 +30,10 @@ set `TEST_PGHOST`, `TEST_PGPORT`, `TEST_PGUSER`, `TEST_PGPASSWORD`, at production infrastructure. Full S3 integration tests skip without both required TEST endpoints. PostgreSQL-only regressions require only the disposable PostgreSQL TEST variables and can run with `python -m pytest -q tests/integration/test_postgres.py`. -Local validation of those regressions plus unit tests passed 43 tests on disposable -PostgreSQL 18.6; the complete RustFS integration suite remains a rollout gate. -Boundary/crash tests inject interruptions at workflow boundaries; actual process -kill/network/disk fault testing and production connectivity remain rollout gates. +Gitea CI passed all 65 unit/integration tests against disposable PostgreSQL 18.0 and +RustFS 1.0.0, then published an immutable image. Boundary/crash tests inject interruptions +at workflow boundaries; actual process-kill/network/disk fault testing remains additional +operator validation. Production pod connectivity was verified on Citadel. `requirements.lock` pins runtime packages and hashes; `requirements-test.lock` adds pytest. Regenerate with Python 3.12, pip 25.2 and pip-tools 7.5.1: @@ -235,7 +235,11 @@ Source INSERT/UPDATE/TRUNCATE/REFERENCES/TRIGGER and ledger DELETE are denied; t role has no permanent object creation privilege, elevated attributes or memberships. A real login/read-only ledger/source-access check succeeded. `.env` now uses this restricted role instead of the application's administrative credentials. No source -rows or indexes were changed. Index/capacity preflight and deployment remain pending. +rows were changed during that setup. During explicitly authorized rollout, valid +`(recorded_at,id)` indexes were built concurrently for the price tables. PostgreSQL +HBA access was added only for `btcprod`/`btcdata_archiver` from Citadel's +`10.42.1.0/24` pod subnet, backed up and reloaded without a restart. No source rows +were changed by these preparation steps. `scripts/rustfs-create-identity.py` is an explicit operator-only provisioning utility, run on RustFS with access to its administrative credential file. It refuses to overwrite @@ -244,7 +248,32 @@ It is not executed by the runtime job. RustFS 1.0.0 rejects bare `Resource: "*"` mixed action families in a single statement; the policy uses separate admin/STS deny statements and S3 wildcard ARNs compatible with the deployed build. -## Explicit operator setup and rollout (remaining steps not executed) +## Kubernetes deployment state + +GitOps `magnus/deploy` main commit `c635511` registers `apps/btcdata-archive` under +namespace `sejs`. Flux successfully reconciled it. CronJob `btcdata-archive` remains +**suspended**, with Monday 00:00 UTC schedule, Citadel affinity, and the specified +resource/storage limits. Existing feature-branch work in the deploy repository was +preserved; rollout was committed from a separate main-based worktree. + +Deployed image (tested CI commit `d93735b`): +`harbor.pmagnus.cloud/btcdata-archive/btcdata-archive@sha256:9d297326b4dc0d997a344b7353ee48257632be8020094a3a29f02b268e9f8a81`. +Secrets `btcdata-archive-credentials` and `btcdata-archive-pull` were created in `sejs` +outside Git, using the restricted PostgreSQL/S3 identities and a project-scoped +Harbor pull-only identity. CI uses a separate project-scoped push/pull robot. + +Validation Job `btcdata-archive-verify-20261003193601` completed successfully on +Citadel using this image, `run --mode verify-only --max-weeks 1`, non-root execution, +read-only root filesystem and the actual work volume. There were no eligible weeks +because the initial catch-up had completed, so it validated schema, lock, database +access, S3 versioning and ledger discovery without uploading or pruning new data. +The full export/upload/exact-version-download/prune/recovery path passed in CI. + +Enabling `suspend: false` is a separate explicit approval/GitOps step. No recurring +archival/pruning schedule was enabled during deployment. Update image digests +explicitly; this app has no floating-tag image automation. + +## Operator setup and rollout reference 1. Run disposable integration tests. Inspect live schemas, constraints/DELETE triggers, RLS, existing indexes, source volumes/per-week sizes, source role privileges, @@ -264,11 +293,11 @@ statements and S3 wildcard ARNs compatible with the deployed build. RustFS IAM compatibility. Runtime can inspect versioning but cannot enable it, administer the bucket, or delete objects/versions. No expiry/lifecycle rules. Provisioning and CI image pushes require separate credentials, never runtime keys. -5. GitOps artifacts are prepared in `../deploy/apps/btcdata-archive/`, **not registered - in the parent apps kustomization**. They remain suspended and have a deliberately - unpullable placeholder digest. Choose a remote, configure CI/Harbor, publish the - tested image and replace that digest before registration. Provision Secrets in - `sejs` through out-of-Git tooling; namespace references cannot cross namespaces. +5. GitOps artifacts are in `magnus/deploy` main at `apps/btcdata-archive/`, registered + in the parent apps kustomization and currently suspended with a tested immutable + digest. For upgrades, publish a tested CI image and update the digest explicitly. + Provision Secrets in `sejs` through out-of-Git tooling; namespace references cannot + cross namespaces. Never commit runtime or registry credentials. 6. Confirm requests/limits/work capacity against the largest old week. Initial work limit is 8 GiB; disk estimates are conservative, not a capacity guarantee. Limit whole-week files require an increase for large weeks; streaming is not parallelism. diff --git a/scripts/test-integration.sh b/scripts/test-integration.sh index a800e46..86a78bb 100755 --- a/scripts/test-integration.sh +++ b/scripts/test-integration.sh @@ -10,7 +10,7 @@ trap '"${compose[@]}" down -v' EXIT runner="$("${compose[@]}" ps -aq tests)" # CI's workspace is inside its job container, not on the Docker host. Copy only # test inputs; never bind-mount the workspace or copy .env, .git, or production secrets. -COPYFILE_DISABLE=1 tar -cf - requirements-test.lock src tests sql \ +COPYFILE_DISABLE=1 tar -cf - pyproject.toml requirements-test.lock src tests sql \ scripts/archive-old-weeks.sh scripts/list-archives.sh scripts/wait-test-infrastructure.py \ | docker cp - "$runner:/app" docker start --attach "$runner"