exzen-core — Staging Environment¶

Status: Deployed ✅ (2026-05-10) URLs:

• Frontend (Mini App): https://telegram-staging.exzentcg.com (Vercel Preview env, pinned to staging branch)
• Backend API: https://telegram-staging-api.exzentcg.com (Proxmox CT 106, gated by WEBHOOK_SECRET only — no Cloudflare Access)

Snapshot: initial-deploy (taken 2026-05-10 after first green E2E order test)

Purpose¶

Hosts a stable staging environment for ExzenTCG/exzen-core — the existing Telegram Mini App + Python FastAPI backend currently running in a long-lived terminal on Oracle Cloud. Staging exists so the operator can test changes (UX tweaks, new features, integration changes) end-to-end before promoting them to the production Oracle host. Also a useful first step toward containerising the backend so the production deployment can eventually move off "runs in a terminal" onto a supervised container.

This is a stopgap until the TCG Commerce Operations Platform replaces exzen-core at Phase 10 go-live. Don't over-invest — the goal is "stop testing in prod" until the replacement lands.

Deployment Details¶

Prerequisites — what you create by hand before running this runbook¶

These are owner-managed (Claude can't edit 00_secrets). Add each to homelab/00_secrets/Credentials Index.md as you go.

1. BotFather — create two new bots:
2. Staging admin bot, e.g. ExzenTCG_Staging_Admin_bot (record token)
3. Staging customer bot, e.g. ExzenTCG_Staging_bot (record token)
4. On the staging customer bot: set Menu Button URL to https://telegram-staging.exzentcg.com once Vercel is wired up
5. Staging admin Telegram forum group:
6. Create a new Telegram group → enable Topics → add both staging bots → make staging admin bot an admin with topic permissions
7. The bot will discover the group via /set (see Step 9 below) — no manual chat ID extraction needed
8. Google Sheet:
9. Duplicate the production master Sheet → rename to ExzenTCG Staging → share with the existing service account email (unchanged from prod)
10. Record the new Sheet ID
11. Secrets to generate:
12. WEBHOOK_SECRET — fresh random string (different from prod)
13. NEXTAUTH_SECRET — fresh random string for the Vercel staging project
14. Vercel — set up Preview + branch override for the staging branch (one-time, in https://vercel.com/exzentcgs-projects/exzentcg):
15. Settings → Domains → Add telegram-staging.exzentcg.com → in the domain row, set Git Branch to staging. This pins this hostname to deploys built from the staging branch only.
16. Settings → Deployment Protection → Vercel Authentication: untick "Require Log In" (or set to "Only Production Deployments" if that option appears). The default Hobby setting gates all Preview deploys behind a Vercel SSO login, which the Telegram webview can't satisfy → Mini App opens to the SSO popup forever. The Mini App enforces its own auth via Telegram initData HMAC and WEBHOOK_SECRET, so this Vercel layer is redundant for this project and breaks the Telegram flow.
17. Settings → Environment Variables: for each variable below, click Add New → Value → Environment = tick Preview → check "Apply to specific Git branches" → enter staging. Repeat for each:
    • BACKEND_API_URL=https://telegram-staging-api.exzentcg.com
    • WEBHOOK_SECRET=<staging webhook secret>
    • CUSTOMER_BOT_TOKEN=<staging customer bot token> (used by /api/auth/telegram to validate Mini App initData HMAC — without it auth fails 500 and the Mini App stays on the access-restricted gate even after Telegram opens the URL correctly)
    • NEXT_PUBLIC_BOT_USERNAME=<staging customer bot username> (e.g. Zencorestore_staging_bot. Used by the admin login page; without it staging falls back to the prod bot handle.)
    • NEXTAUTH_URL=https://telegram-staging.exzentcg.com
    • NEXTAUTH_SECRET=<staging>

!!! danger "Always set the branch override" If you forget the staging branch override on any of these vars, every PR preview in this project will inherit the staging value — including pointing the Mini App at the staging backend. Custom Environments (Pro tier) eliminate this risk; on Hobby it's checklist discipline.

!!! note "Vercel doesn't auto-rebuild on env-var changes" Adding a new env var (or changing one) does not trigger an automatic redeploy. After updating env vars, either push an empty commit to the staging branch or click Redeploy on the latest staging deployment in the Vercel dashboard (untick "Use existing Build Cache" so the new var is baked into the lambda runtime). Without a fresh build, the running deployment serves the old env values.

Deployment Plan¶

Step 1 — Create LXC¶

From the Proxmox host (192.168.0.200):

pct create 106 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname exzen-staging \
  --cores 1 \
  --memory 2048 \
  --swap 1024 \
  --rootfs local-lvm:10 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.0.56/24,gw=192.168.0.1,firewall=1 \
  --nameserver 192.168.0.1 \
  --features nesting=1 \
  --onboot 1 \
  --start 1

Step 2 — Install Docker + set DNS¶

pct enter 106
apt update && apt upgrade -y && apt install curl git -y
curl -fsSL https://get.docker.com | sh
exit

# Fix DNS from Proxmox host (router DNS has known lookup issues — see CT 103, 105)
pct set 106 --nameserver 1.1.1.1
pct reboot 106

Step 3 — Clone exzen-core and prepare config¶

pct enter 106
mkdir -p /opt/exzen-core && cd /opt/exzen-core
git clone https://github.com/ExzenTCG/exzen-core.git .
git checkout staging   # long-lived staging branch — never deploy main/stablev2 here

Create backend/config/.env with staging values (NEVER commit this file — it's gitignored):

cd /opt/exzen-core/backend
mkdir -p config
cp config/templates/.env.example config/.env
nano config/.env

Required staging values (the rest copy from prod via password manager):

# IMPORTANT: ENVIRONMENT=PROD on staging.
# The Settings class derives TELEGRAM_TEST_MODE from ENVIRONMENT==DEV. When
# TELEGRAM_TEST_MODE is true, the bots route their API calls to Telegram's
# *test environment server* (/bot{token}/test) which is a separate Telegram
# instance. Our staging bots are regular @BotFather bots, not test-env bots,
# so they get rejected by the test server. Setting ENVIRONMENT=PROD here
# bypasses that routing. Marketplace integrations are blank on staging so the
# PROD-mode Shopee/Lazada gateways are inert.
ENVIRONMENT=PROD

# Bot tokens — staging-only, from BotFather. Settings reads the *_PROD keys
# when ENVIRONMENT=PROD (NOT plain TELEGRAM_BOT_TOKEN — that name is silently
# ignored thanks to pydantic_settings extra="ignore").
TELEGRAM_BOT_TOKEN_PROD=<staging admin bot token>
CUSTOMER_BOT_TOKEN_PROD=<staging customer bot token>

# Authorised admin user IDs — comma-separated string, NOT JSON array.
# (.env.example shows JSON; that example is misleading — Settings uses str split.)
AUTHORIZED_USER_IDS=<your Telegram user ID>

# Sheets — staging Sheet ID
GOOGLE_SHEET_ID=<staging sheet id>

# Customer Mini App URL — Vercel staging domain
CUSTOMER_MINIAPP_URL=https://telegram-staging.exzentcg.com

# BFF auth — fresh random
WEBHOOK_SECRET=<staging webhook secret>

# Payments — leave blank to disable QR generation (real-money safety guard).
# Once you trust the staging environment and want to test end-to-end PayNow
# flows, you can set a dedicated test mobile/UEN here. WARNING: any QR
# generated then routes real SGD to that recipient.
PAYNOW_MOBILE=
PAYNOW_UEN=
COMPANY_NAME=ExzenTCG (Staging)

# Marketplace integrations — leave blank to disable, or use sandbox/test creds.
# DO NOT use prod marketplace credentials — staging would touch real listings.
PARTNER_ID=
PARTNER_KEY=
SHOP_ID=
NGROK_AUTHTOKEN=

# Sentry — optional, separate project for staging
GLITCHTIP_DSN=

Place the Sheets service account JSON at backend/config/service_account.json (same file as prod — the duplicated Sheet is shared with the same service account).

Step 4 — Build and start the backend¶

The Dockerfile + docker-compose.yml live in the exzen-core repo at backend/ (Python 3.11-slim, non-root uid 1000, named volumes for data/, logs/, orders/, qr_temp/, healthcheck on /health). Confirm the checkout has them before continuing — ls backend/Dockerfile backend/docker-compose.yml should both succeed.

cd /opt/exzen-core/backend
docker compose up -d --build
docker compose ps
docker compose logs --tail 100 exzen-core

for v in exzen-data exzen-logs exzen-orders exzen-qr-temp; do
  chown -R 1000:1000 $(docker volume inspect backend_$v --format '{{.Mountpoint}}')
done
docker compose restart

First-boot checks:

• Backend logs show Uvicorn running on http://0.0.0.0:5000 and Application started.
• Both bots log connection success (admin + customer) — look for getMe HTTP/1.1 200 OK for each token.
• curl http://localhost:5000/health returns 200.
• SQLite migrations run automatically at startup; volume exzen-data now contains the DB file.

docker compose exec -u app exzen-core python -c "from init_db import init; init()"
docker compose restart

Step 5 — LXC Firewall¶

Create /etc/pve/firewall/106.fw on the Proxmox host:

[OPTIONS]
enable: 1

[RULES]
IN ACCEPT -source +edge_gw -p tcp -dport 5000      # Only edge-gateway can reach FastAPI
IN ACCEPT -source +admin_desktop -p tcp -dport 22   # SSH from admin
IN ACCEPT -source +admin_desktop -p tcp -dport 5000 # Direct LAN access for dev
IN DROP                                              # Block all other inbound
OUT ACCEPT -dest +router_gw -p udp -dport 53        # DNS
OUT ACCEPT -dest +router_gw -p tcp -dport 53        # DNS
OUT ACCEPT -p udp -dport 53                          # Public DNS fallback (1.1.1.1)
OUT DROP -dest +lan_subnet                           # Lateral movement prevention
OUT ACCEPT -p tcp -dport 443                         # HTTPS — Telegram, Sheets, Vercel BFF, marketplaces
OUT ACCEPT -p tcp -dport 80                          # HTTP fallback (apt, image registries)

Verify after boot:

# From inside CT 106

# 1. LAN lateral-movement should be blocked
curl -s --connect-timeout 3 http://192.168.0.16 -o /dev/null -w "HTTP %{http_code}\n"
# Expected: HTTP 000

# 2. Proxmox host should be blocked
curl -s --connect-timeout 3 https://192.168.0.200:8006 -o /dev/null -w "HTTP %{http_code}\n"
# Expected: HTTP 000

# 3. FastAPI self-reachability — use loopback (same iptables OUTPUT caveat as CT 105)
curl -s --connect-timeout 3 http://localhost:5000/health -o /dev/null -w "HTTP %{http_code}\n"
# Expected: HTTP 200

# 4. Telegram API reachable (polling mode requires this)
curl -s --connect-timeout 5 https://api.telegram.org -o /dev/null -w "HTTP %{http_code}\n"
# Expected: HTTP 200 or 302

External reachability test — from CT 101 edge-gateway, proves NPM can proxy:

pct enter 101
curl -s --connect-timeout 3 http://192.168.0.56:5000/health -o /dev/null -w "HTTP %{http_code}\n"
# Expected: HTTP 200
exit

Step 6 — Edge-gateway firewall update¶

Edit /etc/pve/firewall/101.fw on the Proxmox host. Add before the OUT DROP -dest +lan_subnet line:

OUT ACCEPT -dest 192.168.0.56 -p tcp -dport 5000   # Proxy to exzen-staging FastAPI

Step 7 — NPM Proxy Host¶

Open NPM admin: http://192.168.0.51:81 → Proxy Hosts → Add:

Step 8 — Cloudflare Tunnel Route¶

Cloudflare → Zero Trust → Networks → Tunnels → exzentcg-homelab → Published application routes → Add:

Cloudflare auto-creates the DNS CNAME (proxied).

Step 9 — First-time bootstrap (admin channel + product cache)¶

Two pieces of state live in the SQLite DB and must be re-seeded after any fresh deploy or volume wipe — bot_settings.customer_order_channel_id and the products_cache rows. Without them the Mini App's /api/products returns 503 Service Unavailable and order notifications have nowhere to land.

1. Bind the admin channel. In the staging Telegram forum group, send /set to the staging admin bot. Tap 🛒 Set Customer Order Channel in the reply — the bot writes the group + thread mapping to bot_settings. Verify by re-sending /set and confirming the configured channel ID is shown.
2. Refresh the product cache. DM /refresh_products to the staging admin bot. The bot pulls the staging Sheet's Master Inventory + Imports tabs and writes ~370 rows into products_cache. Bot replies with a count.
3. (Optional) Configure error log + scheduled report destinations via /set's other buttons.

Step 10 — Vercel staging frontend¶

(Outside the CT — done in the Vercel dashboard.)

1. Create a new Vercel project linked to ExzenTCG/exzen-core, root directory frontend, OR add a staging git branch on the existing project promoted to a custom domain.
2. Add custom domain telegram-staging.exzentcg.com.
3. Set environment variables (Production scope on the staging project / Preview scope for the staging branch):
4. BACKEND_API_URL=https://telegram-staging-api.exzentcg.com
5. WEBHOOK_SECRET=<staging webhook secret> (must match backend)
6. NEXT_PUBLIC_TG_BOT_NAME=ExzenTCG_Staging_bot
7. NEXTAUTH_URL=https://telegram-staging.exzentcg.com
8. NEXTAUTH_SECRET=<staging>
9. Deploy the staging branch.
10. In BotFather, set the staging customer bot's Menu Button URL to https://telegram-staging.exzentcg.com.

Step 11 — End-to-end verification¶

• [ ] https://telegram-staging-api.exzentcg.com/health returns 200 (no Cloudflare Access challenge)
• [ ] https://telegram-staging.exzentcg.com loads in a browser (if you see Vercel SSO instead of the page, recheck Deployment Protection in the prereq)
• [ ] Open the staging customer bot in Telegram on a separate test account, tap the menu button at the bottom of the chat → Mini App opens (opening the URL directly in a browser hits the "must be opened through ExzenTCG miniapp" gate — that's the Mini App's own initData check, working as designed)
• [ ] Mini App home page renders without "Access Restricted" (if it doesn't, the most likely cause is CUSTOMER_BOT_TOKEN missing on Vercel — auth route returns 500 silently. See prereq.)
• [ ] Browse products → add a priced item to cart (only the products with non-zero standard_price in the staging Sheet's Imports tab can be checked out; unpriced items make subtotal=0 and the BFF Zod schema rejects with "Invalid request body") → checkout with Self-Collection (free, safe) → order success page renders
• [ ] If PAYNOW_* is set in .env: in-app QR block renders with the configured recipient. If PAYNOW_* is blank: success page falls through to the legacy "Open Telegram Bot" CTA (safety guard working as designed)
• [ ] Order appears in the staging admin forum group with approval buttons
• [ ] Approving the order in the staging group flips the order status in the staging Mini App
• [ ] Sending a chat message to the staging customer bot from the test account forwards into the order's thread in the staging admin group
• [ ] No data hits the production Sheet (check Sheet revision history)
• [ ] No notifications hit the production admin forum group

Step 12 — Snapshot¶

pct snapshot 106 initial-deploy --description "exzen-staging initial deployment — FastAPI + python-telegram-bot polling mode"

Updates needed elsewhere (post-deploy)¶

• [ ] Add exzen-staging entries to homelab/00_secrets/Credentials Index.md:
• SSH key, root password (CT 106)
• Staging admin bot token (@zencorehelper_staging_bot)
• Staging customer bot token (@Zencorestore_staging_bot)
• Staging WEBHOOK_SECRET
• Staging Google Sheet ID
• Staging admin forum group ID (-1003922201568)
• Vercel staging project + NEXTAUTH_SECRET
• [ ] Update homelab/01_planning/Phase 1/Architecture Diagram.md — add CT 106 to IP map and Mermaid diagram
• [ ] Add CT 106 health check to homelab/02_setup_logs/Phase 1 Health Checks.md (curl backend /health, check docker compose ps)
• [ ] Update Homepage services.yaml on CT 103 with a card for telegram-staging (LAN-internal entry)
• [ ] Add a row to root index.md "Homelab Services" table
• [x] ~~Open a PR on ExzenTCG/exzen-core adding backend/Dockerfile + backend/docker-compose.yml~~ — merged as PR #22
• [ ] Write homelab/02_setup_logs/exzen-staging Setup Log.md while deploying — raw commands, screenshots of NPM/Cloudflare configs, the BotFather setup, and any deviations from this plan

Troubleshooting¶

Rollback¶

If the staging environment goes sideways and you need to start over:

# From Proxmox host
pct stop 106
pct rollback 106 initial-deploy
pct start 106

After rollback, re-run Step 9 first-time bootstrap (/set + /refresh_products) — both pieces of state live in the SQLite DB which initial-deploy snapshot took before you bound the admin channel and refreshed products.

cd /opt/exzen-core/backend
docker compose down
docker volume rm backend_exzen-data backend_exzen-logs backend_exzen-orders backend_exzen-qr-temp
docker compose up -d --build
# then re-run the volume-chown loop (Step 4) and Step 9 bootstrap

If even the snapshot is bad: pct destroy 106 and re-run this runbook from Step 1. Staging is disposable by design — no real data lives here.

Promoting a staging-validated change to production¶

Once a feature has been exercised on this CT (real order through, no regressions in bot delivery, no console errors, etc.), promote it to production:

1. Merge staging → stablev2 on exzen-core¶

# On your local exzen-core checkout
git fetch origin
git checkout stablev2
git pull origin stablev2
git merge --ff-only origin/staging   # fast-forward only — no merge commit unless conflicts
git push origin stablev2

If a fast-forward isn't possible (someone hot-fixed stablev2 directly), resolve by rebasing staging onto stablev2 and force-pushing staging, then re-running the merge.

2. Vercel auto-redeploys the production project¶

Pushing to stablev2 triggers the production Vercel project to rebuild and deploy the new Mini App. No further action needed for the frontend.

3. Restart the Oracle Cloud backend¶

The backend on Oracle does NOT auto-deploy. SSH in and restart manually:

ssh oracle-prod
cd /path/to/exzen-core/backend
git pull origin stablev2
# Stop current process (Ctrl-C in tmux/screen, or kill the uvicorn process)
# Re-activate venv if needed, then:
python main.py

4. Smoke-test production¶

Place a real (small-amount) order from your operator account through the production Mini App. Confirm the QR + bot DM + admin thread + Sheets write all behave as on staging.

5. Tag the release (optional but useful)¶

git tag -a "release-$(date +%Y-%m-%d)" -m "Promoted from staging: <one-line summary>"
git push origin --tags

Tags make rollback trivial: git checkout <previous-tag> on Oracle, restart.