btcdata-archive/PLAN.md

btcdata-archive implementation handoff

Status: IMPLEMENTED LOCALLY — restricted RustFS IAM identity, versioned archive bucket, PostgreSQL ledger and restricted runtime role provisioned on explicit requests; source index preflight, deployment and deletion have not been performed. Implementation artifacts and validation instructions are in README.md. The disposable PostgreSQL/RustFS suite is prepared but requires a working Docker environment before rollout. Prepared: 2026-10-03. Target repository: /Users/pmagnus/pi/btcdata-archive.

1. Objective and agreed decisions

Build a standalone archiver for PostgreSQL database btcprod. Every Monday at 00:00 UTC, archive expired data from three fixed tables into one SQLite database per UTC week in RustFS. Delete source rows only after the uploaded archive has been downloaded and verified. Immediately after successful cleanup, PostgreSQL retains the last four completed UTC weeks plus any rows arriving in the current week.

User decisions:

• Tables: public.strike_price, public.binance_price, public.kraken_minute.
• Choose these tables once; do not dynamically select the largest tables each run.
• Archive only when a week expires, not immediately after that week finishes.
• One SQLite file per week containing all three tables.
• Dedicated RustFS bucket: btcdata-archive; prefix: btcprod/weeks/.
• Keep archives indefinitely. One verified RustFS copy is sufficient before PostgreSQL deletion; independent backups are outside this project.
• Initial catch-up must process all old weeks, oldest first, strictly one week at a time. Provide a dedicated script for this operation.
• Run the recurring archiver in Kubernetes namespace sejs as a CronJob pinned to Citadel and managed through the existing GitOps repository.
• Preserve PostgreSQL data when uploads or verification fail; retry and catch up later.
• Archived data is downloaded and queried manually. No historical-query API or application UI is required.
• At planning time this document was the only authorized deliverable. The subsequent request to implement this plan authorizes local code, tests, dependencies and setup/deployment artifacts, not production provisioning, jobs or source deletion.

2. Evidence and environment

Read-only inspection on 2026-10-03 found these PostgreSQL sizes, including indexes and auxiliary storage:

Table	Estimated live rows	Total storage
strike_price	1,727,101	1,148 MB
binance_price	3,308,108	258 MB
kraken_minute	294,591	35 MB

Estimates are planning evidence, not export counts. Recheck schemas, constraints, privileges, source volume, and free space before implementation rollout.

Relevant existing repositories:

• /Users/pmagnus/pi/btcdata: Clojure application, migrations in resources/migrations, SQL in resources/queries.sql.
• /Users/pmagnus/pi/deploy: GitOps manifests; btcdata currently has namespace btcdata and runs on Citadel or srv1893094.
• /Users/pmagnus/pi/system/source/rustfs-xtdb-implementation.md: infrastructure record for RustFS.

Recorded infrastructure:

• PostgreSQL: Citadel, documented LAN address 192.168.50.195:5432, database btcprod.
• RustFS: documented S3 endpoint http://192.168.50.45:9000, with path-style access; credentials are held outside Git.
• RustFS record describes a 200 GiB data disk and existing bucket xtdb. Use a separate btcdata-archive bucket.
• Endpoint reachability, RustFS versioning behavior, available storage, and current Kubernetes configuration have not been validated for this archiver.
• The existing application holds advisory lock 728194106 for its own writer fencing. The archiver must use a different lock and must not start or contend with the application's writer components.

Read applicable AGENTS.md and skills before touching any repository or host. Never print credential-bearing JDBC URLs, .env contents, or tokens. Git initialization and any remote repository creation are future work; no remote hosting destination has been selected.

3. Retention and week calculation

Use UTC exclusively, independent of host timezone and Europe/Copenhagen daylight saving.

At invocation start:

1. Capture the current UTC instant once.
2. Find Monday 00:00 UTC at the beginning of that instant's week, called anchor.
3. Set cutoff = anchor - 28 days.
4. Keep this cutoff fixed for the invocation, including a long catch-up run.
5. Eligible rows have their retention timestamp strictly less than cutoff.

Weeks are half-open intervals [Monday 00:00, next Monday 00:00). Archive only complete weeks ending at or before the cutoff.

Use recorded_at for Binance and Strike. Use Unix-second ts for Kraken minute candles, not their insertion-time created_at.

Example: invocation on Monday 2026-10-05 has cutoff 2026-09-07 00:00 UTC. In steady state, it archives 2026-08-31 through just before 2026-09-07. It retains 2026-09-07 onward. A row exactly at the cutoff stays in PostgreSQL.

Weekly cleanup means the retained span grows from approximately 28 days to approximately 35 days between successful runs. Failed runs can leave more data. Do not add daily pruning or strict rolling-28-day enforcement.

4. Standalone project layout and runtime

Use Python with pinned dependencies: psycopg 3 for PostgreSQL, boto3 for S3, and Python's standard sqlite3. Use pytest for tests. Pin a supported Python base image and dependency versions when implementing; do not choose floating production image tags.

Proposed layout:

btcdata-archive/
  PLAN.md
  README.md
  pyproject.toml
  <dependency lock file>
  src/btcdata_archive/
    cli.py
    config.py
    weeks.py
    postgres.py
    sqlite_archive.py
    object_store.py
    archive.py
  scripts/
    archive-old-weeks.sh
  sql/
    setup.sql
    indexes.sql
  tests/
    unit/
    integration/
  Dockerfile
  .dockerignore
  .gitignore

Implementation stays in this new repository. Deployment manifests belong in the existing deploy repository, following its instructions. Do not add archiving logic or dependencies to the btcdata application's startup path.

Runtime configuration:

• PGHOST, PGPORT, PGDATABASE=btcprod, PGUSER, PGPASSWORD; support PGSSLMODE as appropriate to the existing database deployment.
• S3_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION=us-east-1.
• ARCHIVE_BUCKET=btcdata-archive, ARCHIVE_PREFIX=btcprod/weeks/.
• ARCHIVE_WORK_DIR=/work, ARCHIVE_BATCH_ROWS=5000.

Fail clearly if required configuration is missing. Source credentials from Kubernetes Secrets, never tracked files or CLI arguments containing passwords. Use boto3 path-style S3 addressing. Do not silently fall back to a public AWS endpoint.

5. CLI and initial catch-up script

Provide these interfaces:

• python -m btcdata_archive inventory: read-only report of anchor, cutoff, oldest eligible week, candidate weeks, and source counts. No bucket writes or source deletion.
• python -m btcdata_archive run --mode verify-only: export and upload all eligible weeks sequentially, verify them, but do not delete source rows.
• python -m btcdata_archive run --mode archive-and-prune: the production workflow over all eligible weeks.
• Both run modes accept optional --max-weeks N for a bounded rollout/test. Omission means all eligible weeks. Reject zero or negative values.
• python -m btcdata_archive status: report last completed week, incomplete runs, and remaining expired source data.

python -m btcdata_archive requires a package __main__.py routing to the CLI.

Required script: scripts/archive-old-weeks.sh

Implement a thin executable wrapper around the same tested CLI used by the CronJob. It must not contain a second implementation of archiving or deletion.

• Require an explicit --mode verify-only or --mode archive-and-prune; no implicit destructive default.
• Forward optional --max-weeks N.
• Read all connection settings from the environment.
• Use strict shell error handling and exec the Python CLI so exit codes and termination signals propagate correctly.
• By default process every eligible old week, starting with the oldest, one at a time.
• Complete export, upload verification, and (in prune mode) source deletion for a week before advancing to the next week.
• Stop immediately on failure or unresolved changed rows. Do not skip a failed week and continue with later weeks.
• Print the fixed cutoff and a progress line for each week: started, verified, pruned, complete, or failed, with table counts.
• Rerunning the script resumes safely from archived objects and PostgreSQL progress records; it must not require a local checkpoint file.
• No parallel week processing, background subprocess exports, or loading the entire backlog into memory/disk.

Discover the earliest eligible timestamp across the three tables, combine this with incomplete ledger entries, and enumerate weeks in ascending order through the cutoff. Create an empty weekly database for an internal gap between eligible weeks so the archive sequence is explicit. Do not create years of empty archives preceding the first actual source week. Recheck for late rows in old weeks on every new invocation.

Suggested future operator usage:

./scripts/archive-old-weeks.sh --mode verify-only --max-weeks 1
./scripts/archive-old-weeks.sh --mode archive-and-prune

These are implementation targets, not commands to run while preparing this document.

6. SQLite schema and object contract

Object key: btcprod/weeks/YYYY-MM-DD.sqlite, where the date is the starting Monday. Example: btcprod/weeks/2026-08-31.sqlite.

Every file includes:

• binance_price: id INTEGER PRIMARY KEY, price TEXT NOT NULL, recorded_at TEXT NOT NULL.
• strike_price: id INTEGER PRIMARY KEY, rates TEXT NOT NULL, recorded_at TEXT NOT NULL, quote_sats INTEGER NULL.
• kraken_minute: ts INTEGER PRIMARY KEY; open, high, low, close, vwap, and volume as non-null TEXT; trade_count INTEGER NOT NULL; created_at TEXT NOT NULL.
• Indexes on (recorded_at, id) for the two price tables.
• Metadata tables containing archive format version, source database/schema, source column types, week start/end, creation time, per-table row counts, and canonical content checksums.

Preserve PostgreSQL numeric values as exact decimal strings; never convert through Python float or SQLite REAL. Preserve all timestamp microseconds using a fixed UTC ISO-8601 representation. Preserve SQL NULL distinctly from JSON null. JSONB's original whitespace/key order is not meaningful; export deterministic JSON preserving exact numeric values. Obtain JSONB as text and canonicalize it without float conversion.

Canonical row checksums use primary-key order and a documented deterministic encoding with explicit nulls and field boundaries (canonical JSON arrays of typed string/integer/null values are suitable). Use SHA-256. Compare the source stream's count/digest to a fresh SQLite read before upload.

Check the live source schema against the supported schema before exporting. An added, removed, or incompatible column must fail closed with a clear migration-required error; never silently omit columns. Future schema changes require an explicit archive format migration and tests.

Use a self-contained SQLite file: commit and close all connections before hashing/uploading. Do not rely on a separate WAL or journal file. Run PRAGMA integrity_check, requiring ok.

7. Safe export, verification, deletion, and recovery

Hold PostgreSQL advisory lock 728194107 on the same live session coordinating the run. If unavailable, exit without starting work. If that session is lost, stop; never silently reconnect and continue deleting. This lock only serializes archivers, not the application's writers.

For each week:

1. Check available working storage. Only keep files needed for the current week, including any downloaded prior archive and verification copy. Abort safely on inadequate space or disk errors.
2. If the object already exists, download its exact version and validate its schema, metadata, and integrity. Preserve all previously archived rows, including rows already deleted from PostgreSQL.
3. Read the source week's rows in a read-only repeatable-read transaction, using server-side cursors and bounded batches. Merge source rows into the local weekly archive by primary key. Current PostgreSQL values supersede that key's earlier archived values; prior object versions retain earlier revisions. Close the source snapshot before network upload.
4. Verify exported source rows against SQLite, including counts, values, and canonical hashes. Validate the full merged archive and recompute metadata.
5. Upload the finalized file. Require bucket versioning to be enabled and capture the returned object version ID. Use SHA-256 in object metadata, but do not treat ETag or metadata alone as proof of content.
6. Download that exact uploaded version into a separate local file. Verify its full SHA-256, integrity, schema, week boundaries, and table checksums. Persist the verified version ID and checksum in the ledger.
7. In verify-only mode, stop processing this week here and record it as verified, not pruned.
8. In prune mode, read bounded batches of rows from the verified downloaded file. Stage each batch in a PostgreSQL temporary table using the source column types. Delete using primary-key matches and null-safe equality of every source column, plus the week's timestamp bounds and fixed cutoff. Use IS NOT DISTINCT FROM comparisons, including typed JSONB and numeric comparisons. Commit each batch independently.
9. Recount eligible source rows for this week. If any remain, record an incomplete/reconciliation-required state, stop the run, and retain all remaining rows. A new invocation merges these rows into a new verified object version before attempting deletion again.
10. Mark the week pruned only after the final source check finds no rows. Remove current-week temporary files and proceed to the next oldest week.

Do not hold table locks or a long write transaction during export/upload. Do not delete by timestamp alone. A row changed after export must fail the content match and remain in PostgreSQL. A late insertion after the final check is discovered on the next invocation.

Durable progress: create schema btcdata_archive and a weeks ledger keyed by week_start, with week_end, format version, status, object key, object version ID, file SHA-256, JSONB row/deletion counts, attempt time, completion time, and last error. Statuses: exporting, verified, pruning, complete, failed. Do not store credentials or entire row payloads in errors.

Recovery rules:

• Crash before verified upload: source stays intact; rebuild or merge on retry.
• Crash after upload but before ledger update: validate the existing object and merge current source rows; never assume ledger absence means the object is empty.
• Crash during deletion: reuse/merge the full archived file, retaining already-pruned records, and continue idempotently.
• Completed ledger entry with new old source rows: reopen that week and reconcile.
• Corrupt or incompatible existing archive: stop and require operator recovery; never overwrite it with a partial source remainder.
• Failed verification or unavailable RustFS: no source deletion for that week.

8. Database preparation and deployment

Future setup SQL must be explicit and operator-run; the runtime job must not create database roles or migrate schemas automatically.

• Add (recorded_at, id) indexes for the two price tables with CREATE INDEX CONCURRENTLY, outside transaction-wrapped migrations. Inspect existing equivalent indexes first.
• Reuse the kraken_minute.ts primary-key index.
• Give the runtime role SELECT and DELETE only on the three source tables, TEMP privilege for staging, and the necessary privileges on its own ledger. No application writer lock or INSERT/UPDATE privilege on source tables is needed.
• Use a bucket-scoped identity for list/get/version-get/put and necessary multipart operations. Enable versioning during provisioning with an administrative identity. Do not grant the runtime identity bucket administration or object-version deletion.
• Preserve bucket objects and all versions indefinitely; do not configure expiry or cleanup of completed archive versions.

Kubernetes defaults:

• Namespace sejs, CronJob name btcdata-archive. Provision the namespace through GitOps if it does not already exist. Create the archiver's Secrets, ServiceAccount, and any namespaced configuration in sejs; Secrets cannot be referenced across namespaces. The existing btcdata application remains in its current namespace.
• schedule: "0 0 * * 1", timeZone: Etc/UTC, concurrencyPolicy: Forbid.
• Pin node affinity to citadel, where both database and RustFS LAN access should be validated.
• Execute the CLI in archive-and-prune mode; same sequential workflow as the catch-up script.
• Job backoffLimit: 3; missed schedules can start within six days (startingDeadlineSeconds: 518400). Each execution computes its own fixed current-week cutoff and catches up all overdue weeks.
• Retain three successful and three failed Job records; restartPolicy: Never; termination grace 60 seconds.
• Initial requests: 100m CPU, 256 MiB memory, 2 GiB ephemeral storage. Limits: 1 CPU, 1 GiB memory, 8 GiB ephemeral storage. Disk-backed /work emptyDir with 8 GiB size limit. Revalidate these against the oldest/largest actual week during preflight; increase before enabling if needed.
• Use a non-root container and an immutable image published by the new repository's CI. Follow the existing deploy repository's approved registry and GitOps conventions.
• Start with suspend: true; enabling is a separate rollout step after validation.

Use structured logs for week boundaries, counts, verification state, deletion progress, durations, and failures. Return nonzero on errors. The ledger and status CLI provide durable status. External email/Slack alerts are outside scope; do not create notification integrations without a separate request.

9. Tests and acceptance criteria

Unit tests:

• Monday boundary calculations, Sunday runs, year/ISO-week crossings, leap dates, and DST independence.
• Exactly-at-cutoff rows remain; exactly-at-week-start rows belong to that week.
• Timestamp column selection differs correctly for candles versus price snapshots.
• Exact decimal round trips, large IDs, null quote_sats, JSON numeric precision, UTC microseconds, empty tables, and deterministic checksums.
• Strict schema mismatch detection and validated CLI arguments.

Integration tests with disposable PostgreSQL and the target RustFS image:

• Multiple weeks export in ascending order into one SQLite file per week, with all three tables.
• Script processes all old weeks, holds a fixed cutoff, and never starts the next week until the current one completes.
• Internal empty week, partial first week, no eligible data, and bounded --max-weeks runs.
• Verify-only mode creates verified archives without deleting source data.
• Upload/download failure, checksum mismatch, SQLite corruption, full disk, and missing privileges preserve unverified source rows.
• Kill at each workflow boundary and restart; previously archived/deleted rows must not disappear from the final archive.
• Concurrent archiver exclusion and lock-session loss.
• Source changes and late arrivals during export/deletion remain unpruned until a later verified reconciliation.
• Restore a downloaded archive into an isolated PostgreSQL schema using its documented type mapping and compare every value/count/hash.
• Existing application latest-price and latest-candle reads remain usable while archival runs.

Success means every deleted row exists in a verified RustFS object version with equivalent values; after a successful stable run, the selected source tables contain no rows older than the fixed cutoff. All other tables and application APIs are unchanged.

10. Rollout sequence and operator documentation

1. Implement and test entirely against disposable infrastructure. Initialize version control only when implementation is authorized; remote creation is separate from this plan.
2. Inspect live schemas/indexes, oldest timestamps, per-week sizes, disk capacity, versioning support, and cluster connectivity without exposing secrets.
3. Provision the bucket/versioning, restricted identities, ledger, and indexes through explicit setup steps.
4. Deploy the CronJob suspended. Run one oldest week in verify-only mode using the same production image.
5. Download that object, verify it independently, and restore into an isolated test database/schema. Confirm exact values and types.
6. Invoke the catch-up script in archive-and-prune mode with no week limit. It must process the entire backlog, oldest first, one week at a time. Monitor source activity and stop on the first failure.
7. Confirm the retention cutoff, archive inventory, checksums, and ledger agree; enable the Monday CronJob through GitOps.
8. Document download/query examples, exact decimal handling, manual invocation, status checks, retry recovery, and how to suspend the CronJob.

Rollback: suspend the CronJob through GitOps and stop any active archiver gracefully. Committed deletions are not undone by rollback. Recovery requires restoring the verified SQLite archives; preserve objects and ledger records. Use normal PostgreSQL vacuum/autovacuum for reusable space. VACUUM FULL, immediate filesystem shrinkage, table partitioning, and redesigning application storage are outside scope.

11. References

• Kubernetes CronJobs: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
• SQLite type behavior: https://sqlite.org/datatype3.html
• Existing application migrations and queries in the btcdata repository are the starting schema reference; live PostgreSQL schema is authoritative during preflight.

12. Next-agent instruction

Read this file fully and inspect applicable repository instructions. If asked only to revise the plan, change only this document. If explicitly authorized to implement, create the standalone project and catch-up script described here, complete automated tests, and prepare deployment/setup artifacts. Do not infer production deployment or source-data deletion authorization from a request to write code. Report any live-schema or infrastructure incompatibility before changing the agreed retention or archive contract.