tcg-platform¶

Current role¶

tcg-platform is the staging replacement platform. It is not yet production authority.

The platform should replace Google Sheets/Apps Script, pieces of exzen-core, and parts of the operational finance/inventory stack with a PostgreSQL-backed Medusa system.

Architecture¶

Monorepo components:

• apps/server: Medusa 2 backend (Yarn 4 workspace).
• apps/dashboard: Next.js operator dashboard (OpenNext build, deployed to Cloudflare Workers).
• apps/storefront: Next.js storefront — a standalone npm app, not a root Yarn workspace member (it installs from its own package-lock.json).
• Root Yarn workspaces are apps/server, apps/dashboard, and packages/*. The only package under packages/ today is packages/shared-types (shared TypeScript types); there are no separate API-client, connector-SDK, UI, lint, or TS-config packages.
• PostgreSQL (postgres:16-alpine) for the intended commerce/finance source of truth.
• Redis (redis:7-alpine) with AOF persistence and a named redis-data volume; it backs the Medusa event bus, workflow engine, cache, and locking.
• Custom modules for inventory/imports, DCA, connectors, finance, profitability, RBAC, audit, durable operations, and storefront projection.
• Compose project is pinned to name: tcg-platform so volumes/networks are stable regardless of which directory runs docker compose (prevents drift to server_*-prefixed volumes on pre-monorepo hosts).

Durable operations¶

Runs on the durable outbox spine (Lane C) inside the Medusa process — not a separate worker service. MEDUSA_WORKER_MODE: shared runs the HTTP server and background worker/subscribers in one container (docker-compose.yml).

• Model: append-only outbox (ops_outbox_event), operation runs (ops_operation_run), and attempts (ops_operation_attempt) with DB-clock lease fencing, idempotency keys, and causal attempt outcomes. Failed events land in the exception queue for operator review; replay tooling can re-run a stored raw event without duplicating side effects.
• Execution: the poll-driven drain (src/jobs/ops-worker-drain.ts) runs every 10s: it reclaims stale leases, leases ready outbox events, and drives each through append STARTED attempt → executor.execute → finalize (run-attached) or execute → complete/requeue (standalone). An immediate ops.enqueue.kick subscriber fires right after enqueue, so events are picked up without waiting for the next 10s tick.
• Executors:
• internal — the default tracer executor (no provider I/O). OPS_EXECUTOR only ever accepts internal, so the default can never silently become a provider write.
• inventory-sync — the deliberate provider-writing executor for inventory.sync.requested events: it pushes a variant's CURRENT stock to its Shopee/Lazada listings with desired-state semantics (quantity re-read at execution time). Routing is per event type via resolveEventExecutor (in both the drain job and the kick subscriber), not env-selectable.
• Inventory changes are enqueued on the outbox per (variant, channel) from the inventory.inventory-level.updated event; a 1-second bucket (bucket:<floor(ms/1000)>) coalesces bursts so one run/event carries the batch, and availableAt = now + 1s guarantees every absorbed change is committed before the executor's desired-state read.
• Operator API: GET /admin/ops/{status,outbox,runs,attempts} (view_exceptions) and POST /admin/ops/probe (manage_connectors), which enqueues an internal tracer operation to prove the loop live.
• Tuning env (all optional):

Authentication (as of 2026-09-18)¶

BetterAuth is the credential + sign-in engine; Medusa remains the token and authorization source (connect.sid sessions, authenticate(...) middleware, staff.user_role). BetterAuth is mounted at /better-auth/* with its own ba_* tables (same Postgres).

• Staff: BetterAuth-first dashboard login (loginWithBetterAuth: enrol → link → exchange → session); the legacy emailpass flow is the forced-fallback recovery path.
• Customers: email+password sign-in on the storefront, BetterAuth-first for linked identities with the legacy bridge as fallback.
• Telegram: a first-class BetterAuth provider (mini-app initData + Login-widget OIDC), flipped on staging behind TELEGRAM_BA_LOGIN with silent fallback. Second door: Telegram-only customers can add an email+password (verification email via Resend) and sign in on the web with the same account and order history.
• Email: transactional email (verification, resets) via Resend hooks; suppressed and logged when RESEND_API_KEY is unset.
• Gotchas encoded in code: BetterAuth's CSRF guard rejects origin-less sign-in POSTs (it is NOT browser-only), and session-gated calls must present the resolved cookie name + SIGNED value. Read apps/server/src/lib/better-auth/ before touching auth.

Staging runtime snapshot¶

As of 2026-08-13 (post Ampere migration):

• Host: OCI ampere VM (VM.Standard.A1.Flex, 4× ARM64 Neoverse-N1, 24 GiB RAM, 100 GB disk, Ubuntu 24.04, tailnet 100.85.99.41). Staging previously ran on Proxmox CT105 tcg-staging; that container was destroyed after migration.
• Stack: Docker Compose project tcg-platform at /srv/tcg-platform/apps/server — Medusa, storefront, PostgreSQL, Redis. Images are native arm64 builds.
• Data: PostgreSQL restored from the pre-migration dump; products 186, variants 385. Redis was recreated empty during the one-time migration, but it now runs with AOF persistence (--appendonly yes) and a named redis-data volume, so routine restarts no longer wipe in-flight state; transient queues/cache regenerate only after a fresh volume.
• Ingress: dedicated Cloudflare tunnel exzentcg-ampere (systemd cloudflared-ampere on Ampere) routes tcg-staging.exzentcg.com → 127.0.0.1:9000 and shop-staging.exzentcg.com → 127.0.0.1:8000. tcg-staging sits behind Cloudflare Access; shop-staging is public. The old exzentcg-homelab tunnel and the NPM proxy hosts that used to forward to CT105 no longer carry these hostnames. The tracked apps/server/docker-compose.yml declares 9000:9000 and 8000:8000 (all host interfaces); Ampere's local checkout carries a host-only edit to 127.0.0.1:9000:9000 / 127.0.0.1:8000:8000 and marks that file skip-worktree. This is a host-local security prerequisite, not a tracked Compose invariant: after any fresh checkout or deploy, verify the rendered bindings (for example, with docker compose config or docker ps) before exposing the tunnel.
• Shopee and Lazada connector profiles exist; marketplace inventory sync toggles are off. The provider-writing durable inventory-sync executor exists, but the per-connector sync_enabled gate remains the safety boundary.

Runtime configuration¶

• Required secrets: compose requires JWT_SECRET and COOKIE_SECRET via ${VAR:?} (in apps/server/.env) — a fresh/default deployment cannot boot with a known static secret. Production boot (NODE_ENV=production) additionally refuses to start without REDIS_URL; the event-bus/workflow/cache/locking modules must use real Redis.
• Cookie scope: set COOKIE_DOMAIN=.exzentcg.com on the server side (medusa-config.ts cookieOptions); the dashboard re-emits the same cookie from its login action and restates the scope with its own SESSION_COOKIE_DOMAIN (apps/dashboard/wrangler.jsonc vars, resolved in lib/session-cookie.ts). Two processes, two env vars, one value — change one, change the other. Unset in dev; browsers reject an explicit Domain= for localhost.
• Credential boundary: Shopee/Lazada credentials and OAuth/return URLs come from the host env (.env/systemd EnvironmentFile), never committed; postgres creds come from the compose service env so deploy-server.sh needs no stored passwords.

Staging is active development, not a stable production mirror.

CI/CD¶

GitHub Actions, defined in .github/workflows/. The required status check aggregate accepts only success or path-filter skipped — job cancelled (manual cancel, timeout) is treated as a failure.

CI (.github/workflows/ci.yml)¶

• Triggers: pull requests to main and stacked worktree branches wt/**, plus pushes to main. Superseded runs on the same ref are cancelled; a cancelled run is never validation on its own.
• Path classification: plain git diff (no third-party action) computes server, dashboard, storefront, docs, and ci outputs; heavy lanes only run when their path set changed, but the aggregate check always runs so every PR keeps one stable required check.
• Runner trust boundary: same-repository HUMAN pull requests and push-to-main run on the persistent self-hosted dev-ci runners (persistent Yarn + local BuildKit caches); fork and bot (e.g. Dependabot) PRs stay on GitHub-hosted ubuntu-latest, because a self-hosted runner executes PR-controlled code on a persistent machine.
• Server lane (build): server unit tests (always-on gate), the Medusa build, tsc --noEmit (after the build generates .medusa/types), and a tcg-platform:ci Docker image build (BuildKit cache split by runner type).
• Self-hosted Docker cleanup: after the smoke build's load: true step, the server lane best-effort runs docker image prune -af and docker builder prune -af --filter "until=24h". This removes per-run unused image copies and stale internal BuildKit state; both commands are self-hosted-only and ignored on failure because the four dev-ci runners share a rootless daemon and cleanup can race another build. The persistent /opt/gha-cache/buildkit layer cache is deliberately untouched; hosted runners self-clean and skip this step.
• Dashboard lane (dashboard): dashboard unit tests, then exactly one Next + OpenNext build (opennextjs-cloudflare build), packaged as dashboard-opennext.tar.gz and — on main pushes only — published to the runner-host hand-off for that run: /opt/tcg-ci-artifacts/<run-id>/ holding the tarball, a sha256sum file written inside that directory, and a manifest.txt with sha=<commit> / run_id= / ref=. No GitHub artifact storage is involved — it blocked every dashboard deploy while its org quota figure was stale, and it is now off the deploy path entirely. This remains the sole build.
• E2E: the per-PR CI lane has no e2e job. dashboard-e2e lives in the opt-in/nightly Full regression workflow, builds its own dashboard (yarn workspace @tcg/dashboard build, then test:e2e), and uploads only its Playwright report — it does not consume the deploy bundle.
• Redis/BullMQ integration (integration-redis): runs the real ops-outbox integration suite against Postgres + Redis service containers (event-bus-redis enabled, durable-ops delivery/retry/fencing) whenever the server lane is touched.
• The classifier's storefront output gates a storefront job that installs, lints, type-checks, and runs the storefront unit suites (test:stock, test:freshness, test:auth, test:unit). It does not compile/build the storefront — do not rely on storefront-compile CI coverage today.

Weekly container smoke (.github/workflows/container-smoke.yml)¶

Monday 03:15 UTC, or manual workflow_dispatch, 60-minute budget, hosted runners. Builds the Medusa image, boots fresh Postgres + Redis, runs the full migration graph against the vendored Yarn release, then boots Medusa and requires /health = 200. Deliberately not a per-PR gate (fresh-DB migrations are minutes-long); per-PR migration safety comes from the migration unit suites and the deploy script's preflight/migration/rollback.

Staging access & deploy¶

• Reach the host: tailscale ssh ubuntu@100.85.99.41 (repo at /srv/tcg-platform).
• Runtime path note: Ampere's checkout is /srv/tcg-platform, and the script's default REPO_ROOT/COMPOSE_DIR follow that checkout. The script still defaults BACKUP_DIR to /opt/backups/tcg-platform and DEPLOYED_SHA_FILE to /opt/tcg-platform/SERVER_DEPLOYED_SHA; set those environment variables explicitly if the host's /srv layout is canonical for backups and deployment records.

Server deploy (manual)¶

scripts/deploy-server.sh, run ON Ampere. It is a manual, pinned-commit procedure — there is no auto-deploy on push for the server:

1. Lock (single deploy at a time) + fetch, then pin an exact SHA ([SHA] arg or origin/main).
2. Candidate-compose preflight: validates the compose file of the tree being switched TO (read from git, never the current tree) before anything changes.
3. Full PostgreSQL backup (custom-format pg_dump → gzip, integrity + sha256 checked) with retention, default to /opt/backups/tcg-platform/.
4. Switch to the pinned SHA; optional migration preflight (--check-migrations) runs post-switch/pre-up and rolls back on failure.
5. docker compose up -d --build medusa — rebuilds only the medusa service; Redis/Postgres/storefront service-spec changes need a separate explicit Compose apply.
6. Real /health wait; on any post-switch failure (build, health) it rolls back to the previous SHA and prints the DB-restore instruction. Records the deployed SHA.

Options: [SHA], --check-migrations, --dry-run, --help.

• Build order trap: start Medusa first, then build the storefront — the storefront next build collects page data from Medusa on localhost:9000 and fails with ECONNREFUSED if Medusa is not up.
• Verification: on-host curl 127.0.0.1:9000/health → 200; public https://shop-staging.exzentcg.com/sg → 200; https://tcg-staging.exzentcg.com/connectors/shopee → 404/410 means Medusa is reachable through the Access-exempt path.

Dashboard deploy (automatic after green CI)¶

Staging dashboard deploys to Cloudflare Workers via the deploy-dashboard-staging workflow — not a direct push trigger:

• Fires on successful push-to-main CI completion (workflow_run), checks out the exact head_sha CI validated, and promotes the bundle CI published to /opt/tcg-ci-artifacts/<the triggering run id>/ — verifying directory + files exist, that manifest.txt carries exactly that head_sha (grep -qx "sha=<sha>"), and that the checksum passes, before anything is extracted. No rebuild fallback: every miss refuses loudly. No second build in deploy.
• A path filter (dashboard, shared types, the workflow, yarn.lock, package.json) can skip deployment; workflow_dispatch is the manual fallback, which builds once because there is no CI artifact to promote.
• Runs on the self-hosted dev-release runner with the staging environment and CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID secrets.

• Hand-off precondition: it is host-local by construction — every dev-ci runner and the dev-release runner live on the same machine, so the deploy lane reads what CI wrote. If the deploy runner ever moves to another host, the hand-off must move with it (shared mount) or the promotion refuses. Retention is 2 days, pruned by the publishing step; a re-run publishes a fresh directory for the same run id, and the manifest's sha= line is what proves which commit a directory holds.

• Storefront deploy: auto on main pushes (deploy-storefront-staging workflow, trusts successful main CI); manual pinned path scripts/deploy-storefront.sh <SHA> on Ampere. The server remains manual (deploy-server.sh).

• CI: lean merge lane per push + opt-in/nightly Full regression; mandatory ci/full surfaces and the landing process live in the repo's docs/CI-REVIEW-GATE.md. Merges run through scripts/land-pr.mjs (exact-head pinned).

The staging server/dashboard deploys ship staging only; production is the co-hosted tcg-platform-prod stack described above (api-tcg.exzentcg.com + shop.exzentcg.com + dashboard.exzentcg.com). It still must not become the marketplace stock writer until exzen-core is retired — see the boundary below.

What exists now¶

tcg-platform already has strong foundations for:

• Product/variant/inventory models.
• Purchase orders, imports, fees, allocations, receipts, and batch lifecycle.
• DCA/consolidation with provenance and cost conservation.
• Shopee inbound event/idempotency architecture.
• Durable operations spine: outbox/run/attempt on DB-clock leases, retry/fencing, exception queue with operator review UI, raw event store, and replay/reconciliation tooling.
• Connector exception surface and alerting.
• Order COGS allocation and profitability.
• Finance statement/CSV/P&L logic.
• RBAC/audit framework.
• Dashboard operator workflows.
• Dashboard navigation and pages: a single Products surface with Catalog + Stock tabs (stock console with per-channel drift lives under the Stock tab; the old /inventory page redirects there).
• Opening-stock migration tooling and reconciliation.

Production (co-hosted on Ampere, live 2026-09-23)¶

Production is a second Compose stack on the same host as staging, not a separate machine. The stack identity, ports and every derived path come from one knob per stack (see apps/server/docker-compose.yml + scripts/deploy-server.sh).

• Config is per-stack .env. TCG_STACK_NAME, MEDUSA_HOST_PORT, STOREFRONT_HOST_PORT, STOREFRONT_BUILD_MEDUSA_URL, STOREFRONT_IMAGE and the CORS/trusted-origin vars (STORE_CORS, ADMIN_CORS, AUTH_CORS) are read from that stack's apps/server/.env. The deploy scripts refuse a shell-only or disagreeing value for the identity knobs, and derive the health probe, provenance marker, backup directory and rollback file from the resolved stack — so a co-hosted deploy cannot borrow another stack's provenance or backups. LOCK_FILE is deliberately shared (one deploy at a time per host).
• First boot of a stack is manual (/srv/tcg-platform-prod/bootstrap-prod.sh server|storefront <sha>): the deploy scripts require a provenance marker plus a running container to prove the rollback target, so a fresh stack is booted once by hand and its marker is then written from the running image's own revision label.
• Production started empty (no data copied from staging): region Singapore/SGD, the Default Sales Channel, one stock location, a publishable key carrying STOREFRONT_PUBLISHABLE_KEY from the prod .env, and one admin user (admin@exzentcg.com; credential held by the owner). Catalog import is a separate go-live step.
• Cloudflare: dedicated Access apps exzen-tcg-prod (+ /connectors/* and /better-auth/verify-email|reset-password bypass apps) and exzen-tcg-dashboard-prod (+ -static bypass), a exzen-tcg-dashboard-worker-prod service token, and two ingress rules added to the existing exzentcg-ampere tunnel. shop.exzentcg.com is public, like staging's storefront.
• Deliberately dark in production v1: marketplace connectors (no live credentials placed; exzen-core remains the active stock writer) and Telegram (needs its own bot + OIDC client). Transactional email (Resend) is wired.
• BetterAuth on a fresh DB: after the first boot, run medusa exec ./src/scripts/better-auth-migrate.ts inside the container and restart medusa — otherwise every /better-auth/* request 500s with "Database schema mismatch".
• Production dashboard deploys deliberately: deploy-dashboard-production (dispatch-only — sha + a typed confirm) promotes the bundle CI published to the runner-host hand-off for that exact SHA's CI run; it never builds on the release runner, deploys the production worker config only, probes the Access-gated custom domain with a Cloudflare Access service token (never following redirects) and requires the worker's workers.dev route to answer non-200, and posts a release-lifecycle-production record. It refuses a SHA that is not on main, that has no successful push-to-main CI run, or whose hand-off directory is missing or already pruned (2-day retention). First real run 2026-09-25: it promoted 46fc7aa8 (CI run 36011004861, the same run the staging lane had consumed) to worker version fbe1ae4d, smoke 200, and posted the record.
• Bring-up gaps, updated 2026-09-28: both dashboards track main through their lanes — staging auto-promoted 236c56ec (worker b4cc8fdb, 2026-09-25); production promoted 236c56ec (worker 00b95cd9) and then 1c0e05b8 (PR #671, worker f739bdfa, 2026-09-28), each posting a release-lifecycle-production record. The prod server stack was caught up to 24bc45a3 on 2026-09-28 (deploy-prod.sh server <sha>): the script took its own pre-deploy dump (/opt/backups/tcg-platform-prod/medusa-20260928T155048Z-pre-24bc45a3….sql.gz), ran the pending migration scripts, and passed its health probe; the provenance marker reads 24bc45a3. The state-model slice that came with it (S4 backfill / S5 cleanup) needed no data work on prod — both scripts' dry runs report orders scanned=0 / overrides total=0, so nothing was backfilled or deleted. Prod also now ships orders state-model S4+S5 and the purchasing PO-list bulk actions (#663/#664/#668). Production's dashboard is now Access-only — verified live: dashboard.exzentcg.com/login 302s unauthenticated and tcg-dashboard.exzensg.workers.dev/login answers 404 (it answered 200 unauthenticated before this deploy). Note for the lane: the workers.dev check has no propagation grace window, so the first post-#671 run failed on Cloudflare's lag seconds after the deploy while the route was already gone a minute later; its record was superseded by the re-dispatch's success. A retry window on that check is an open follow-up. Production's earlier manual build (d209cf41) is long superseded. Still open: the storefront is browse-only until a public /store/* path exists (same shape as staging). Product images are all hot-linked from marketplace CDNs — measured 2026-09-28: staging holds 1267 image rows, none on the local file provider (829 cf.shopee.sg, 242 on Lazada's test CDN sg-test-11.slatic.net, 185 sg-live.slatic.net, 10 filebroker-cdn.lazada.sg, 1 legacy r2.dev URL), and production holds none yet. Nothing is therefore lost when the local store is wiped on rebuild, but the hot-links are outside our control — the Lazada test CDN especially. The durable path is live in staging as of 2026-09-28: bucket tcg-product-images behind images.exzentcg.com (S3 write and public read verified), the compose passes all six S3_* vars through, and staging's .env now sets them (pre-change backup kept, mode 600 preserved; medusa recreated with all six keys visible inside the container and healthy on :9000). Production is deliberately left unwired until a real upload through staging has survived a rebuild — a proof that still needs an admin session, and no admin credentials are stored in BWS. Also open: the optional mirror of the 1267 hot-linked image URLs into the bucket, and the same .env wiring on production.

Public store API (added 2026-09-28)¶

api-tcg.exzentcg.com/store/* is public via the Access app exzen-tcg-prod-store (53f11a69-f77c-491a-bf8c-f5522518b7c3, policy Store API Bypass, decision=bypass). The storefront's SSR never needed it — it reads Medusa over the internal docker network — but the browser-side flows do: customer auth (/store/auth/better-auth/*), carts, and the Telegram Mini App. Everything else on the API stays owner-only (verified: /admin/products, /health, /storeX decoy and / all still 302).

One Free-plan rate-limit rule guards the anonymous write paths: POST /store/carts and POST /store/customers → block, 20 req / 10s per IP, 10s mitigation (ruleset 5aedbafb85b74975acbfaebd5e14e09a). Verified by load test: 28–37 × 429 per 60 parallel writes, reads unaffected.

Rollback records live in ~/workspace/tcg-prod/access-changes/.

Cloudflare Free-plan rate-limit traps¶

Found by making each mistake against the live API: the entrypoint PUT rejects kind; characteristics must include cf.colo.id; mitigation_timeout must be exactly 10; managed_challenge is not entitled; matches (regex) needs a paid plan; and starts_with is accepted at validation but silently never matches — a rule that looks correct and enforces nothing. Always prove a rule with a real load test rather than reading its expression.

Historical incident notes¶

2026 Shopee connector audit and staging incidents records lessons from the former CT 105 staging environment; it is not a current deploy guide.

15 June 2026 — Orders spreadsheet view lessons (from wiki PR #57, not a current phase status):

• Medusa generated listers and query.graph defaulted to 15 rows. Batched DCA allocations/adjustments/batch-items, Shopee escrows, and variant lookups could silently truncate, understating COGS or dropping fees/metadata. Page to exhaustion or set an explicit take sized for the requested keys; small fixtures do not exercise this boundary. The DCA economics implementation pages its listers, while order enrichment sets explicit limits.
• query.graph reads sorting from pagination.order, not a top-level orderBy (silently ignored when cast to any). Use a multi-key order with a unique id tiebreaker for stable offset pagination; see the Orders route and its graph pagination.

Current production boundary¶

Do not treat tcg-platform as the source of truth yet.

In particular, do not enable marketplace stock sync while exzen-core remains the active stock writer — now doubly important, because tcg-platform has a live provider-writing inventory-sync executor (durable ops, above) that would contend with exzen-core for the same marketplace listings.