Record validated suspended Kubernetes rollout and CI setup
This commit is contained in:
@@ -3,9 +3,9 @@
|
||||
Standalone Python archiver for the three fixed `btcprod` tables: `public.binance_price`,
|
||||
`public.strike_price`, and `public.kraken_minute`. **A restricted RustFS IAM identity
|
||||
and versioned archive bucket, progress ledger and restricted PostgreSQL role have
|
||||
been provisioned; source index preflight and deployment are still pending.**
|
||||
An operator run verified the first weekly archive, then hit a ledger transition bug
|
||||
before deletion. That bug is now fixed; deployment is being prepared suspended.
|
||||
been provisioned. The Kubernetes CronJob is deployed through GitOps, suspended.**
|
||||
The initial ledger transition bug is fixed; the operator completed 28 backlog weeks.
|
||||
Rollout validation ran verify-only and did not delete any source rows.
|
||||
The repository is hosted at https://git.pmagnus.cloud/magnus/btcdata-archive;
|
||||
Gitea Actions and project-scoped Harbor CI/pull credentials are configured.
|
||||
|
||||
@@ -30,10 +30,10 @@ set `TEST_PGHOST`, `TEST_PGPORT`, `TEST_PGUSER`, `TEST_PGPASSWORD`,
|
||||
at production infrastructure. Full S3 integration tests skip without both required
|
||||
TEST endpoints. PostgreSQL-only regressions require only the disposable PostgreSQL
|
||||
TEST variables and can run with `python -m pytest -q tests/integration/test_postgres.py`.
|
||||
Local validation of those regressions plus unit tests passed 43 tests on disposable
|
||||
PostgreSQL 18.6; the complete RustFS integration suite remains a rollout gate.
|
||||
Boundary/crash tests inject interruptions at workflow boundaries; actual process
|
||||
kill/network/disk fault testing and production connectivity remain rollout gates.
|
||||
Gitea CI passed all 65 unit/integration tests against disposable PostgreSQL 18.0 and
|
||||
RustFS 1.0.0, then published an immutable image. Boundary/crash tests inject interruptions
|
||||
at workflow boundaries; actual process-kill/network/disk fault testing remains additional
|
||||
operator validation. Production pod connectivity was verified on Citadel.
|
||||
|
||||
`requirements.lock` pins runtime packages and hashes; `requirements-test.lock`
|
||||
adds pytest. Regenerate with Python 3.12, pip 25.2 and pip-tools 7.5.1:
|
||||
@@ -235,7 +235,11 @@ Source INSERT/UPDATE/TRUNCATE/REFERENCES/TRIGGER and ledger DELETE are denied; t
|
||||
role has no permanent object creation privilege, elevated attributes or memberships.
|
||||
A real login/read-only ledger/source-access check succeeded. `.env` now uses this
|
||||
restricted role instead of the application's administrative credentials. No source
|
||||
rows or indexes were changed. Index/capacity preflight and deployment remain pending.
|
||||
rows were changed during that setup. During explicitly authorized rollout, valid
|
||||
`(recorded_at,id)` indexes were built concurrently for the price tables. PostgreSQL
|
||||
HBA access was added only for `btcprod`/`btcdata_archiver` from Citadel's
|
||||
`10.42.1.0/24` pod subnet, backed up and reloaded without a restart. No source rows
|
||||
were changed by these preparation steps.
|
||||
|
||||
`scripts/rustfs-create-identity.py` is an explicit operator-only provisioning utility,
|
||||
run on RustFS with access to its administrative credential file. It refuses to overwrite
|
||||
@@ -244,7 +248,32 @@ It is not executed by the runtime job. RustFS 1.0.0 rejects bare `Resource: "*"`
|
||||
mixed action families in a single statement; the policy uses separate admin/STS deny
|
||||
statements and S3 wildcard ARNs compatible with the deployed build.
|
||||
|
||||
## Explicit operator setup and rollout (remaining steps not executed)
|
||||
## Kubernetes deployment state
|
||||
|
||||
GitOps `magnus/deploy` main commit `c635511` registers `apps/btcdata-archive` under
|
||||
namespace `sejs`. Flux successfully reconciled it. CronJob `btcdata-archive` remains
|
||||
**suspended**, with Monday 00:00 UTC schedule, Citadel affinity, and the specified
|
||||
resource/storage limits. Existing feature-branch work in the deploy repository was
|
||||
preserved; rollout was committed from a separate main-based worktree.
|
||||
|
||||
Deployed image (tested CI commit `d93735b`):
|
||||
`harbor.pmagnus.cloud/btcdata-archive/btcdata-archive@sha256:9d297326b4dc0d997a344b7353ee48257632be8020094a3a29f02b268e9f8a81`.
|
||||
Secrets `btcdata-archive-credentials` and `btcdata-archive-pull` were created in `sejs`
|
||||
outside Git, using the restricted PostgreSQL/S3 identities and a project-scoped
|
||||
Harbor pull-only identity. CI uses a separate project-scoped push/pull robot.
|
||||
|
||||
Validation Job `btcdata-archive-verify-20261003193601` completed successfully on
|
||||
Citadel using this image, `run --mode verify-only --max-weeks 1`, non-root execution,
|
||||
read-only root filesystem and the actual work volume. There were no eligible weeks
|
||||
because the initial catch-up had completed, so it validated schema, lock, database
|
||||
access, S3 versioning and ledger discovery without uploading or pruning new data.
|
||||
The full export/upload/exact-version-download/prune/recovery path passed in CI.
|
||||
|
||||
Enabling `suspend: false` is a separate explicit approval/GitOps step. No recurring
|
||||
archival/pruning schedule was enabled during deployment. Update image digests
|
||||
explicitly; this app has no floating-tag image automation.
|
||||
|
||||
## Operator setup and rollout reference
|
||||
|
||||
1. Run disposable integration tests. Inspect live schemas, constraints/DELETE triggers,
|
||||
RLS, existing indexes, source volumes/per-week sizes, source role privileges,
|
||||
@@ -264,11 +293,11 @@ statements and S3 wildcard ARNs compatible with the deployed build.
|
||||
RustFS IAM compatibility. Runtime can inspect versioning but cannot enable it,
|
||||
administer the bucket, or delete objects/versions. No expiry/lifecycle rules.
|
||||
Provisioning and CI image pushes require separate credentials, never runtime keys.
|
||||
5. GitOps artifacts are prepared in `../deploy/apps/btcdata-archive/`, **not registered
|
||||
in the parent apps kustomization**. They remain suspended and have a deliberately
|
||||
unpullable placeholder digest. Choose a remote, configure CI/Harbor, publish the
|
||||
tested image and replace that digest before registration. Provision Secrets in
|
||||
`sejs` through out-of-Git tooling; namespace references cannot cross namespaces.
|
||||
5. GitOps artifacts are in `magnus/deploy` main at `apps/btcdata-archive/`, registered
|
||||
in the parent apps kustomization and currently suspended with a tested immutable
|
||||
digest. For upgrades, publish a tested CI image and update the digest explicitly.
|
||||
Provision Secrets in `sejs` through out-of-Git tooling; namespace references cannot
|
||||
cross namespaces. Never commit runtime or registry credentials.
|
||||
6. Confirm requests/limits/work capacity against the largest old week. Initial work
|
||||
limit is 8 GiB; disk estimates are conservative, not a capacity guarantee. Limit
|
||||
whole-week files require an increase for large weeks; streaming is not parallelism.
|
||||
|
||||
Reference in New Issue
Block a user