A-01 — Jijibora Deployment (Administrator Operations)

Scope: What runs where for a Jijibora deployment — service, role, and stack topology; single-cell vs multi-cell operation; pilot scaffolding in Soft posture; secrets and infrastructure baseline; what to watch in steady state; and the Soft — pilot sandbox banner rule.

Status: as of 2026-11-09

Blueprint sources: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md; 03 Deploy Topology; A-02 — Runbooks (step-by-step recovery); Data Fabric — Message Bus and Engine Interfaces (Developer Concept Doc); docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md; docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md

What this means for you

You run one cell per community on the standard stack — SurrealDB as the record, Agent BFF as the gate, NATS as the nervous system — and you refuse on doubt rather than letting the cell guess (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md). While the deployment is in Soft posture you run it as a pilot sandbox with manual reconciliation, not as production billing or an SLA (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). If any check below is ambiguous, stop and refuse the action — never substitute a synthetic success (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

Details

1. What runs where — service, role, and stack topology

A Jijibora cell reuses the platform spine. The table below is the operator inventory — keep it pinned to your cell and do not add bespoke tables for a community (source: 03 Deploy Topology).

Operator rules that apply to every row: every successful governed transaction writes a PoCE receipt (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md); on infra failure the gateway fails closed with HTTP 502/503 and never returns synthetic OPTIMAL/APPROVED (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Fail closed everywhere — Camel/Keycloak/Surreal/NATS/Solver failure → 502/503 (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

2. Deployment modes in operators' language

One cell ("Node-in-a-Box"). Agent/Truth BFF, local SurrealDB, local NATS, and optional Redis run as a single autonomous unit. The node governs its own traffic locally and calls out only when policy or residency mandates it (source: 03 Deploy Topology). Provisioning is a Docker Compose stack spinning up the BFF, local SurrealDB, local NATS, and optional Redis on an isolated bridge network (source: 03 Deploy Topology).

Three cell modes — same codebase, no cross-node database linking (source: 03 Deploy Topology):

• dedicated_instance — one organization owns the whole database server. Plain meaning: the community owns its store outright (source: 03 Deploy Topology).
• shared_namespace — several organizations share one server, separated by namespace. Plain meaning: multiple communities share hardware-operations but never share rows (source: 03 Deploy Topology).
• federated_peer — a peer node in a mesh; data stays local, calls go through BFF proxy or NATS leaf (source: 03 Deploy Topology).

Where things live. Placement bindings say where each entity's state lives: organizations.home_cell_id (primary residency), sovereign_account.cell_id (ledger + manifest), implementation_manifest placement, billing_anchor cells; the session carries the resolved active_cell_id and the BFF sets the namespace/database per request (source: 03 Deploy Topology).

Multi-cell (federation) in plain terms. Discovery goes through a registry mapping (org, contract) to a peer BFF; queries and subscriptions go through BFF HTTP proxy plus NATS leaf subjects prefixed by cell; identity federates globally via user_identity (source: 03 Deploy Topology). What never federates: cross-namespace SELECTs, cross-cell SQL joins, shared writable manifests, and raw cross-cell vault database reads — scoped-token proxy API only (source: 03 Deploy Topology). Whole-record allow/deny plus event log is the federation boundary; field redaction is not shipped (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Operator rule: if the record could hold PHI/PII, refuse to federate it until redaction ships (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).

Current limits you must plan around: GAP-DEP02 is partial (Surreal deployment_cell table deferred; deployment_config.cells + realm fields are the record), GAP-DEP05 (full cross-cell proxy) is still open, GAP-DEP03/04/07 are partial, GAP-DEP08 (migration tooling) is deferred; GAP-DEP01 and GAP-DEP06 are largely closed (source: 03 Deploy Topology). The Caddy :8088 and Gravitee :8089 gateway cards rest on May 2026 documents and must be re-verified before being relied on for any new deployment (source: 03 Deploy Topology).

3. Pilot scaffolding — Soft posture and what "go-live" means operationally

Soft = pilot sandbox. The November 2026 record notes a CFG-ED Soft go-live (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Operate every Jijibora pilot in Soft posture:

• Payments are sandbox only (sandbox M-Pesa/Stripe); sell the pilot with manual reconciliation and do not promise auto-debit (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).
• Storage is a single RocksDB file with local MinIO; sell a single-tenant cell with hourly encrypted snapshots as roadmap, not as an SLA (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).
• Tenant onboarding is manual (30–60 min/customer) until POST /v1/admin/tenants/provision ships; offer white-glove onboarding until then (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).
• Do not promise customer-managed keys, field redaction, FIPS validation, BAA/SOC 2, or money-transmitter custody — each is a documented not-built item (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).

What "go-live" means operationally is the four-step lifecycle: Deploy OS → Register node (scope ad, directory, join, trust) → Develop app (VOA + manifest v2 + STP) → Register app (federation stubs + commercial gates) (source: 03 Deploy Topology). Full local schema deploy runs through database/deploy.ps1 with function re-apply via database/deploy-functions.ps1 (source: 03 Deploy Topology). Production runs the same compose file plus the production overlay with an env file (--env-file docker/.env.production), the listed secrets, and restart: unless-stopped; there is no second codebase — only secrets, env, and restart policy change (source: 03 Deploy Topology).

Go-live gate (reject-first). Refuse to declare go-live until all of the following hold: the cell's deployment_mode and placement are recorded; the secrets baseline in §4 passes §4 checks; the Soft banner rule in §6 is accepted by the community owner in writing; and the pilot success criteria are pre-agreed (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). If any item is in doubt, refuse go-live and stay in Soft (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

4. Secrets and infrastructure baseline (Keycloak / OpenBao / MinIO)

• Keycloak :8081 is the identity provider (OIDC, realm-per-tenant, audience mapper, JWKS) (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Down = login/session fail (source: 03 Deploy Topology).
• OpenBao :8200 holds secret values in KV v2 (secret/data/{category}/{path}) with metadata in the secret_store table; rotation is logged in secret_rotation_log (source: 03 Deploy Topology). It is vendor-operated today and is not a customer policy-signing root; never answer "customer-managed keys: yes" until the customer-root check passes (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).
• MinIO :9000/:9001 holds object storage (media, uploads, vault attachments) (source: 03 Deploy Topology).
• Production overlay expects named secrets (Keycloak DB, Synapse DB, Novu/Kutuma, FormIO, and per-service proxy NATS passwords/keys) and runs with restart: unless-stopped (source: 03 Deploy Topology).
• Readiness surface: GET /v1/platform/production-readiness with a secrets template at database/seeds/deployment/production_secrets.example.env; local dev may use DEV_SKIP_AUTH=1 or Bearer dev_<user> (source: 03 Deploy Topology).
• Secret discipline: no container secrets in code, no filesystem reads of JSON/YAML at runtime (runtime reads NATS KV only), no hardcoded price catalogs, no synthetic fallback responses (source: 03 Deploy Topology).

Operator check before any pilot: confirm GET /v1/platform/production-readiness is green on the live cell, confirm Keycloak and OpenBao are reachable on :8081/:8200, and confirm MinIO buckets for the cell exist (source: 03 Deploy Topology). If any is unreachable, refuse to open the pilot — fail closed (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

5. What to watch — health, readiness, outbox depth, consumer receipts, UIC flows

Health / readiness. The Agent BFF boots with no schema and no NATS; its /health reports degraded and the HTTP API stays up (source: 03 Deploy Topology). Claims readiness is GET /v1/claims/processing/readiness — surreal up plus a zero check on the claims path that stays dark when relay/consumer is broken (source: A-02 — Runbooks (step-by-step recovery)). Watch both on every shift; dark + degraded = refuse new governed traffic until fixed (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

Outbox depth. The outbox is the durable delivery anchor: producers CREATE nats_outbox, the relay publishes to NATS JetStream, and each row flows pending → sent → delivered → failed → dead-letter with bounded retries (source: Data Fabric — Message Bus and Engine Interfaces (Developer Concept Doc)). Operators watch:

# Pending depth — must fall to 0 and stay 0
SELECT count() AS c FROM nats_outbox WHERE status='pending' GROUP ALL;

Expected output: {"c": 0} steady-state. If depth never drops, follow R-01 in the runbook below (source: A-02 — Runbooks (step-by-step recovery)). Relay publishes are per-subject bounded (SELECT ... WHERE status='pending' LIMIT 100) so depth, not log volume, is the signal (source: Data Fabric — Message Bus and Engine Interfaces (Developer Concept Doc)).

Consumer receipts. Lifecycle events arrive on the bus and a consumer validates against the pack allowlist, dispatches the guarded transition once per event, and records refusals as failed outbox rows (source: Data Fabric — Message Bus and Engine Interfaces (Developer Concept Doc)). Watch: status='delivered' = published; status='failed' = guard refused and recorded; status='dead-letter' = max retries hit with an audit row created (source: Data Fabric — Message Bus and Engine Interfaces (Developer Concept Doc)). A transition that returns 200 but writes no outbox row is a fault — see R-03 (source: A-02 — Runbooks (step-by-step recovery)).

UIC / identity flows. May 2026 shipped UIC/BBS+ waves 10–30 and federation (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Current headless surfaces are /v1/auth/ (Keycloak + BBS+) and /v1/health/ (ZTELS leases) (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md) over vault-guarded traversal (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). PoCE covers governed paths that start a PoCE session — not every login/payment/sensor (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Operator rule: do not treat a login as proof of a governed action; verify the governance_path_receipt fields instead (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Never label server clock (time::now()) as trusted time (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).

6. The "Soft — pilot sandbox" banner rule

Rule (reject-first). While any of the following remain true — sandbox-only payments, manual provisioning, single-file store — every operator surface that touches money, custody, or uptime must carry the Soft — pilot sandbox treatment and the pilot must be sold with manual reconciliation and no auto-debit promise (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Specifically:

• Refuse to remove the banner or claim production billing until the live payment cutover ships (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).
• Refuse to present single-file storage + local MinIO as an SLA; present hourly encrypted snapshots as roadmap (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).
• Refuse to present receipts as dispositive in litigation, the vendor as system of record for regulatory filing, or the organization as certified because it bought software (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).
• Refuse customer-held-root language; use "vendor-operated OpenBao today" until the customer-root check passes (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

The banner is cleared only when the same gates that make Soft necessary are closed and the community owner re-signs the go-live gate in §3 (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). On doubt — banner stays (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

Runbook / playbook checkpoints (for A-02 style structure)

RUNBOOK D-01 — Outbox relay stalls / pending depth stays high

Symptom: [BFF] Platform outbox relay failed: Error: Cannot perform subtraction with 'outbox:...' and 'NONE' every 15s, or GET /v1/claims/processing/readiness shows surreal up but zero check on the claims path stays dark; pending depth (SELECT count() AS c FROM nats_outbox WHERE status='pending' GROUP ALL;) never drops (source: A-02 — Runbooks (step-by-step recovery)).

Cause (proven 2026-11-09): bare sqlString built mark_outbox_published(outbox:01a...) (parsed as arithmetic) and recordOutboxFailure was unguarded, aborting the sweep (source: A-02 — Runbooks (step-by-step recovery)).

Do:

# 1. Confirm depth
# (run the depth query from §5) — expected: c > 0 and not falling
# 2. Drain on a TEMP instance only (never drain production by hand)
$env:PLATFORM_OUTBOX_SUBJECT_PREFIXES='claims.lifecycle'
node src/index.js   # from apps/agent-bff with DEV_SKIP_AUTH=1
# 3. Verify
node scripts/verify-claims-event-consumer-live.mjs

Expected output: consumer-live script passes; depth returns to {"c": 0} (source: A-02 — Runbooks (step-by-step recovery)). Refuse rule: never build function args from bare sqlString — use sqlText for string values (source: A-02 — Runbooks (step-by-step recovery)). On doubt, refuse to mark rows delivered by hand (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

RUNBOOK D-02 — Transition returns 500 / "Unknown function"

Symptom: POST /v1/claims/processing/:claimId/transition returns 500 with Unknown function 'array::some' (source: A-02 — Runbooks (step-by-step recovery)).

Cause: SurrealDB 3.0.5 does not ship array::some; membership check is $x IN $arr (source: A-02 — Runbooks (step-by-step recovery)).

Do:

surreal import --endpoint http://127.0.0.1:8000 --ns africa --db community_os --user root --pass root database/schema/strata/s282_processing_claim.surql
# re-run the transition — expected: 200 with outbox.ok true

Expected output: 200 and outbox.ok true. General rule: exercise any stratum function before trusting it on the live plane (source: A-02 — Runbooks (step-by-step recovery)). On doubt, refuse the transition (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

CHECKPOINT D-03 — Relay not started at boot

Symptom: [BFF] Platform outbox relay not started: await is only valid in async functions… — a require-time error killed the module (source: A-02 — Runbooks (step-by-step recovery)).

Do: node --check apps/agent-bff/src/nats-outbox-relay.js — expected: no output (exit 0). Every helper used in string building is synchronous; every fetch function that awaits is async (source: A-02 — Runbooks (step-by-step recovery)). Refuse rule: if the check fails, do not start the BFF — fix first (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

CHECKPOINT D-04 — Camel routes missing

Symptom: claimgen intake/extgate silently absent; readiness camel probe may still be green (source: A-02 — Runbooks (step-by-step recovery)).

Do: rebuild the image after env changes (docker compose -f docker/compose.microservices.yml --profile compliance) and prove end-to-end with the round-trip script, never /q/health alone (source: A-02 — Runbooks (step-by-step recovery)). Claims Camel routes load only with CLAIMGEN_CAMEL=1 (source: A-02 — Runbooks (step-by-step recovery)). Refuse rule: if the round-trip fails, refuse gated dispatches — Camel down = 502/503 (source: 03 Deploy Topology).

CHECKPOINT D-05 — :3030 predates new routes

Symptom: new endpoints 404 on the shared BFF while temp ports work (source: A-02 — Runbooks (step-by-step recovery)).

Do: either restart :3030 (coordinate) or verify on a temp port (PORT=3071, DEV_SKIP_AUTH=1) and note it (source: A-02 — Runbooks (step-by-step recovery)). Refuse rule: do not claim a route is live until it succeeds on :3030 (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

CHECKPOINT D-06 — Stale function after a stratum edit

Symptom: DEFINE FUNCTION via HTTP returns "NotAllowed"; surreal import can silently drop function bodies (source: A-02 — Runbooks (step-by-step recovery)).

Do: re-import and spot-check the function (RETURN fn::x::y(...) echo) before relying on it (source: A-02 — Runbooks (step-by-step recovery)). Refuse rule: on any doubt about function freshness, refuse to run governed traffic until the echo passes (source: docs/jijibora-gtm/08_SECURITY_AND_COMPLIANCE_POSTURE.md).

CHECKPOINT D-07 — Soft gate (go-live / banner clearance)

Symptom: request to clear the Soft banner or declare production billing/SLA.

Check: live payment cutover shipped? Automated provisioning shipped? Multi-file store + supported HA shipped? (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md). Do: require written owner sign-off against each item plus GET /v1/platform/production-readiness green (source: 03 Deploy Topology). Expected output on pass: all three items true and readiness green. Refuse rule: if any item is false or unverified, refuse clearance — banner stays, manual reconciliation continues (source: docs/jijibora-gtm/00_7_MONTH_TECHNICAL_AUDIT.md).