03 Deploy Topology

Scope: Running stack inventory with ports as documented, cell/namespace modes, gateway layers, and secrets/ops notes. Excludes program status (see 01) and plane architecture (see 02).

Status: As of 2026-11-09. Ports are taken as stated in the sources below, not a live port scan for this doc set; compose files were skimmed for names/ports/services only. DEPLOYMENT_TOPOLOGY v1.0 is dated 2026-05-18 and PLATFORM_MASTER_PLAN v2.0 dated 2026-05-19.

Blueprint sources:

• docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md
• docs/truth-graph/PLATFORM_MASTER_PLAN.md
• docker/compose.microservices.yml
• docker/compose.production.yml
• docs/commercialization/02_HEADLESS_FIRST_CROSS_SECTIONAL_ARCHITECTURE.md
• docs/commercialization/10_ONE_PLATFORM_COMPOSITION.md
• docs/platform-map/A-02-RUNBOOKS.md
• AGENTS.md

How to read this doc. The system of record (SoR) is the place that counts as truth for a piece of data. A BFF (Backend For Frontend) is the gateway software that serves an app's API; it reads the SoR and talks to other services but does not own the data. A cell is one self-contained deployment unit (its own database, gateway, and message bus). A namespace (NS) is a database partition inside one server; a database (DB) sits inside a namespace. Fail closed means a dependency being down produces an explicit refusal, never an invented success.

Local stack inventory

Ports are documented values from the sources below, not a live port scan for this doc set (source: AGENTS.md).

(Source: AGENTS.md; docker/compose.microservices.yml; docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md)

Other services present in compose (not in the core inventory above)

How the stack boots

A node is provisioned as a Docker Compose stack (e.g. docker-compose.sovereign-runtime.yml) spinning up the BFF, local SurrealDB, local NATS, and optional Redis on an isolated bridge network (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §3.1). Full local schema deploy runs through database/deploy.ps1 (function re-apply via database/deploy-functions.ps1) (source: AGENTS.md).

Single-cell vs multi-cell, in plain terms

• One cell ("Node-in-a-Box"). Agent/Truth BFF, local SurrealDB, local NATS, and optional Redis run as a single autonomous unit. The node governs its own traffic locally and calls out only when policy or residency mandates it (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §§1–3).
• Cell modes. Each cell's deployment_mode is one of three values: dedicated_instance (its own SurrealDB), shared_namespace (several organizations share one server, partitioned by namespace), or federated_peer (a peer node in a mesh). The same codebase serves all three placements, with no cross-node database linking (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §4).
• Where things live. Placement bindings say where each entity's state lives: organizations.home_cell_id (primary data residency), sovereign_account.cell_id (ledger + manifest), implementation_manifest placement (where the manifest is stored), billing_anchor cells (commerce may centralize on the control plane). A session carries the resolved active_cell_id and the BFF sets the namespace/database per request (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §4.1).
• Multi-cell (federation). Discovery goes through a registry mapping (org, contract) to a peer BFF; queries and subscriptions go through BFF HTTP proxy plus NATS leaf subjects prefixed by cell; identity federates globally via user_identity. What never federates: cross-namespace SELECTs, cross-cell SQL joins, shared writable manifests, and raw cross-cell vault database reads (scoped-token proxy API only) (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §§6.1–6.4).
• Reference layout. Tanzania: cell:gov_tz (dedicated), cell:exchange_tz (shared namespaces per org), cell:met_tz (dedicated), plus a control plane with federation registry and entitlements (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §8). Network lifecycle runs deploy OS → register node → develop app → register app, with Wave 0 placing deployment_cell, deployment_config, placement resolution, registry stub, and decisions (source: docs/truth-graph/PLATFORM_MASTER_PLAN.md §2a).

Current limits

GAP-DEP02 is partial (Surreal deployment_cell table deferred; deployment_config.cells + realm fields are the record), GAP-DEP05 (full cross-cell proxy) is still open, GAP-DEP03/04/07 are partial, GAP-DEP08 (migration tooling) is deferred; GAP-DEP01 and GAP-DEP06 are largely closed (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §11).

Cell modes at a glance

Gateway story

Three gateway layers exist in the sources; treat the oldest as possibly stale (see honesty note).

1. The unified gateway is the Agent BFF on :3030.

External callers use POST /v1/soko/gateway (capability, action, integration, payload) resolved against the interface catalog; downstream failure is fail-closed with HTTP 502/503 and a PoCE receipt on success (source: docs/commercialization/02_HEADLESS_FIRST_CROSS_SECTIONAL_ARCHITECTURE.md §3). The Agent BFF also proxies all /v1/dde/* to the Truth BFF (default http://127.0.0.1:3031) (source: AGENTS.md).

1. Caddy on :8088 (docker/truth-graph-gateway) forwards to the Truth BFF at host.docker.internal:3031, with smoke script gateway-truth-smoke.ps1 (source: AGENTS.md).
2. Gravitee nginx on :8089 is a dev gateway forwarding /api/v1/dde/*, with smoke script gravitee-truth-smoke.ps1 (source: AGENTS.md).

Honesty note (stale risk). The federation and master-plan documents are dated May 2026 (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md; docs/truth-graph/PLATFORM_MASTER_PLAN.md), and the topology doc points to a September 2026 realm-fabric design (docs/deployment/realms/) that was not read for this file — so the :8088/:8089 gateway cards above are quoted from the older documents and must be re-verified against the realm design before being relied on for any new deployment (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md header; docs/platform-map/A-02-RUNBOOKS.md).

Secrets follow the same rule everywhere: secret values live in OpenBao KV v2 (secret/data/category/path) with metadata in the secret_store table, tenant material is encrypted with tenant- and user-specific keys, and no container secrets, runtime filesystem price reads, or synthetic fallback responses are allowed (source: AGENTS.md; docs/commercialization/10_ONE_PLATFORM_COMPOSITION.md).

Production-readiness status is GET /v1/platform/production-readiness with a secrets template at database/seeds/deployment/production_secrets.example.env; local dev may use DEV_SKIP_AUTH=1 or Bearer dev_<user> (source: AGENTS.md).

Production overlay

Production runs the same compose file plus the production overlay with an env file (--env-file docker/.env.production), the listed secrets, and restart: unless-stopped (source: docker/compose.production.yml). There is no second codebase for production — only secrets, env, and restart policy change (source: docker/compose.production.yml).

Failure posture: what widens vs. what degrades

• Widens (single point of failure): SurrealDB. It is the system of record for all domain state on :8000. Down = nothing can be read or written. This is accepted by design: every node carries its own local SurrealDB so one cell's outage does not take down peers (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §§1–3).
• Widens (single point of failure): Agent BFF :3030. It is the one unified gateway; other BFFs (Truth :3031, bridges) can be down without blocking the claims path, but a :3030 outage closes the front door (source: docs/commercialization/02_HEADLESS_FIRST_CROSS_SECTIONAL_ARCHITECTURE.md §3; docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §3.1).
• Degrades (does not die): NATS. The Agent BFF boots with no schema and no NATS; its /health reports degraded and the HTTP API stays up (source: AGENTS.md). What breaks without NATS is event flow: outbox relay stops publishing, event consumers go quiet — stored state remains readable from SurrealDB (source: docs/platform-map/A-02-RUNBOOKS.md R-01).
• Degrades (fail closed, by decision): solver, Camel, Kestra, Keycloak, OpenBao. Solver down refuses sourcing solves (503); Camel down refuses gated dispatches (502/503); Kestra unreachable blocks timer enforcement; Keycloak down blocks auth; OpenBao down blocks secret resolution. Every one of these is a deliberate refuse, not a silent success (source: AGENTS.md; docs/commercialization/37_CLAIMS_ORCHESTRATION_BACKLOG.md; docs/platform-map/A-02-RUNBOOKS.md R-05).
• Degrades (does not die): Redis, Superset, Gotenberg, Plane, observability. Cache, reporting, document conversion, tracking, and metrics are off the critical path (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md §3.1; AGENTS.md).

Ops checklist

After any route change in the Agent BFF:

• [ ] R-07 — :3030 predates new routes. Symptom: new endpoints 404 on the shared BFF while temp ports work. Rule: after shipping routes, either restart :3030 (coordinate with the other agent) or verify on a temp port (PORT=3071, DEV_SKIP_AUTH=1) and note it (source: docs/platform-map/A-02-RUNBOOKS.md R-07).
• [ ] R-01 — Outbox relay. If pending depth (SELECT count() AS c FROM nats_outbox WHERE status='pending' GROUP ALL;) never drops, use sqlText for outbox ids and per-row guarded failure-recording; drain a stuck queue fast on a temp instance with PLATFORM_OUTBOX_SUBJECT_PREFIXES='claims.lifecycle' (source: docs/platform-map/A-02-RUNBOOKS.md R-01).
• [ ] R-02 — Transition 500s. Unknown function 'array::some' means SurrealDB 3.0.5 incompatibility: re-import the stratum and confirm idempotency before trusting it (source: docs/platform-map/A-02-RUNBOOKS.md R-02).
• [ ] R-05 — Camel routes missing. Claims routes load only with CLAIMGEN_CAMEL=1; prove with the round-trip script, never /q/health alone, and rebuild the image after env changes (source: docs/platform-map/A-02-RUNBOOKS.md R-05).
• [ ] R-08 — Stale function after stratum edit. DEFINE FUNCTION via HTTP returns NotAllowed; surreal import can silently drop function bodies, so re-import and spot-check the function before relying on it (source: docs/platform-map/A-02-RUNBOOKS.md R-08).

Facts

• Core stack per AGENTS.md environment section: SurrealDB v3.0.5 at http://127.0.0.1:8000, NATS :4222, Redis :6379, Agent BFF :3030, Truth BFF :3031, Materra Bridge :3032, Kazi Connector :3047, Keycloak :8081, MinIO :9000/:9001, OpenBao :8200, Security Redis :6380, Superset :8090 with Superset PG :5435, Prometheus :9090, Grafana :3035, OTel :4317/:4318, Fleetbase, Gotenberg :3094, Plane :8091, Appwrite BFF :3050, Kazi Backend :21210 / Site :5173 / Recruiter :5174, compliance-camel :8181 (profile compliance), Kestra :8180, Solver Gateway :3070 (source: AGENTS.md).
• Local identity per AGENTS.md environment: SurrealDB NS africa, DB community_os, root/root auth, rocksdb:database/data-fresh file store; deploy via database/deploy.ps1 (full) and database/deploy-functions.ps1 (function re-apply) (source: AGENTS.md).
• Sovereign API split per AGENTS.md: Edge API apps/sovereign-api :3040 (Vercel) for auth, tenants, vault health; Runtime BFF apps/agent-bff :3030 (Docker) for NATS, dialog sessions, attestation, truth proxy, STP; Console apps/sovereign-console :10300 (Vercel) SPA with VITE_API_URL and optional VITE_AGENT_BFF_URL (source: AGENTS.md).
• Compose microservices per docker/compose.microservices.yml (skimmed for services/ports): fleetbase 9002:9002 with healthcheck /v1/health, fleetbase-bridge (depends on fleetbase + nats, NATS_URL nats://nats:4222), minio 9000:9000 / 9001:9001, minio-init (creates synapse-media, formio-uploads, vault-attachments, afiyalink-phi-dev, vault-doc-library, sovereign-backup buckets), keycloak-db and synapse-db postgres:16-alpine, mongo:7, redis:7-alpine, keycloak 24.0, novu api/web/ws (ghcr.io/novuhq/novu), compliance-camel:latest, message-interceptors (source: docker/compose.microservices.yml).
• Production overlay per docker/compose.production.yml: overlay on docker/compose.microservices.yml with --env-file docker/.env.production; expects secrets KEYCLOAK_DB_PASSWORD, SYNAPSE_DB_PASSWORD, NOVU_DB_PASSWORD / NOVU_API_KEY, FORMIO_DB_PASSWORD / FORMIO_API_KEY, KEYCLOAK_PROXY_NATS_PASSWORD, SYNAPSE_PROXY_NATS_PASSWORD / SYNAPSE_SHARED_SECRET, NOVU_PROXY_NATS_PASSWORD, KUTUMA_PROXY_NATS_PASSWORD / KUTUMA_API_KEY, and runs with restart unless-stopped (source: docker/compose.production.yml).
• Gateway facade per doc 02 section 3: Agent BFF :3030 is the single unified gateway; external callers use POST /v1/soko/gateway (capability, action, integration, payload) resolved against cns_interface_catalog; fail-closed on downstream failure with HTTP 502/503 and PoCE receipt on success (source: docs/commercialization/02_HEADLESS_FIRST_CROSS_SECTIONAL_ARCHITECTURE.md).
• Truth proxy per AGENTS.md Agent BFF section: agent-bff :3030 proxies /v1/dde/* to Truth BFF :3031 (default http://127.0.0.1:3031); sovereign-console SPA is reached via http://localhost:10300 (IPv6-only bind) (source: AGENTS.md).
• Truth gateways per AGENTS.md truth-graph ops: Caddy gateway docker/truth-graph-gateway :8088 forwards to BFF host.docker.internal:3031; Gravitee nginx dev gateway :8089 forwards /api/v1/dde/*; smoke scripts gateway-truth-smoke.ps1 and gravitee-truth-smoke.ps1 (source: AGENTS.md).
• Kutuma gate per AGENTS.md Kutuma section: Keycloak + oauth2-proxy at :3052 fronts the SPA while :3059 serves direct web without auth; Keycloak 24 requires KC_HOSTNAME=host.docker.internal + KC_HOSTNAME_PORT=8081 with provider choice keycloak-oidc (source: AGENTS.md).
• Multi-tenant scoping per doc 02 section 5: single-host multi-tenant topology on Docker/Dokku/Kubernetes with shared SurrealDB :8000, NATS :4222, Keycloak :8081, OpenBao :8200, Camel :8181, Solver :3070, Agent BFF :3030, MinIO :9000; tenant isolation via SurrealDB scopes (CNS-typed), NATS account callouts with scope.<tenant_id>.* subject prefixes, Keycloak realms, and MinIO buckets (source: docs/commercialization/02_HEADLESS_FIRST_CROSS_SECTIONAL_ARCHITECTURE.md).
• Autonomous node definition per DEPLOYMENT_TOPOLOGY sections 1–3: Node-in-a-Box is the fundamental deployment unit (Agent/Truth BFF, local SurrealDB, local NATS, optional Redis cache) provisioned as a Docker Compose stack; node governs its own traffic locally and calls out only when policy or residency mandates it (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Cell modes per DEPLOYMENT_TOPOLOGY section 4: deployment_cell.deployment_mode is one of dedicated_instance, shared_namespace, or federated_peer; the same codebase serves dedicated, shared, and federated placements with no cross-node DB linking (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Placement bindings per DEPLOYMENT_TOPOLOGY section 4.1: organizations use home_cell_id, organization_virtual_app uses runtime_cell_id (defaulting to org home), sovereign_account uses cell_id, implementation_manifest uses implementation_context.deployment_cell_id, billing_anchor uses primary_cell_id (+ optional billing_cell_id); session carries active_cell_id resolved from org + account + config and BFF sets USE NS/DB per request (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Federation mechanics per DEPLOYMENT_TOPOLOGY sections 6.1–6.4: discovery via registry mapping (org_id, contract_uri) to peer BFF + manifest_id; queries and subscriptions via BFF HTTP proxy plus NATS leaf subjects prefixed peer.{cell_id}.; identity federates globally via user_identity; explicitly NOT federated are cross-namespace SELECTs, cross-cell SQL joins, shared writable semantic_manifest across cells, and raw cross-cell vault DB reads (proxy API with scoped token only) (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Config and runtime resolution per DEPLOYMENT_TOPOLOGY section 5: versioned deployment_config (example cfg:tz:national_exchange:v3) with cells, routing (manifest_discovery registry:control_plane, cross_cell_calls bff_proxy, feed_federation nats_leaf) and feature flags; runtime resolves fn::placement::resolve(org, account, config_version) to cell, endpoint, namespace, and bff_base_url; hot reload drains pools and swaps routing without schema migration (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Tanzania reference layout per DEPLOYMENT_TOPOLOGY section 8: cell:gov_tz (dedicated), cell:exchange_tz (shared NS with ns_org_*), cell:met_tz (dedicated), control plane with federation registry plus Stripe/entitlements; deployment waves DEP-W1 through DEP-W6 (cell + config, BFF placement.resolve, registry + publish sync, POST /v1/federation/proxy/{contract_uri}, NATS prefix + leaf config, 3-cell demo) (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Network lifecycle per PLATFORM_MASTER_PLAN section 2a: Deploy OS → Register node (scope ad, directory, join, trust) → Develop app (VOA + manifest v2 + STP) → Register app (federation stubs + commercial gates), with first-party apps AaaS, AfiyaLink, MARS, and Jamii Online; Wave 0 (W0-1–W0-4) places deployment_cell, deployment_config, placement.resolve, registry stub, and ADRs (source: docs/truth-graph/PLATFORM_MASTER_PLAN.md).
• Secrets platform per AGENTS.md APF section and doc 02 section 5: secret values live in OpenBao KV v2 (secret/data/category/path) with metadata in the secret_store table; tenant medical/identity material is encrypted with tenant-specific and user-specific OpenBao keys; secret rotation via secret_rotation_log (source: AGENTS.md).
• Runtime secret discipline per doc 10 section 1 and doc 02 section 3: no container-level secrets in code, no .json/.yaml reads from filesystem at runtime (runtime reads NATS KV buckets only), no hardcoded BPS/price catalogs in routes or UI, no synthetic fallback responses (source: docs/commercialization/10_ONE_PLATFORM_COMPOSITION.md).
• Production-readiness surface per AGENTS.md Agent BFF section: production secrets template at database/seeds/deployment/production_secrets.example.env with status GET /v1/platform/production-readiness; local dev may use DEV_SKIP_AUTH=1 or Bearer dev_user tokens (skip auth otherwise requires control-plane middleware) (source: AGENTS.md).

Open items / gaps

• Ports are documented values, not a live port scan for this doc set; ~30 containers running is AGENTS.md's stated environment, not a verified count here (source: AGENTS.md).
• Federation gap register per DEPLOYMENT_TOPOLOGY section 11: GAP-DEP01 largely closed, GAP-DEP02 partial (Surreal deployment_cell table deferred — deployment_config.cells + realm_id/owner_cell_id are SoR), GAP-DEP03 partial (registry + publish gate exist, cross-broker mesh ops open), GAP-DEP04 partial, GAP-DEP05 still open for full proxy, GAP-DEP06 largely closed, GAP-DEP07 partial, GAP-DEP08 deferred greenfield migration tooling (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Thin coverage: realm-fabric 2026-09 pointer (docs/deployment/realms/) is cited by DEPLOYMENT_TOPOLOGY header but its docs were not part of the sources read for this file; Vercel deployment caveat (public Surreal required for sovereign-api :3040) is stated in AGENTS.md without a live URL (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md).
• Compose skim note: only service names, images, and ports were taken from docker/compose.microservices.yml and docker/compose.production.yml; internal networks, volumes, and healthcheck probe semantics were not audited (source: docker/compose.microservices.yml).
• Gateway staleness risk: Caddy :8088 and Gravitee :8089 cards rest on May 2026 documents; the September 2026 realm-fabric design may supersede them (source: docs/truth-graph/DEPLOYMENT_TOPOLOGY_AND_FEDERATION.md; docs/platform-map/A-02-RUNBOOKS.md).