Design — Enforcing Fail-Closed / No Synthetic Fallbacks (GAP-03)

Related Gap: GAP-03 — Infrastructure errors returning synthetic success instead of failing closed AGENTS.md Mandates: Mandate 3 (fail-closed: "Any call to an external or internal infrastructure service MUST FAIL CLOSED (HTTP 502 / 503 / 422 or throw)"), Mandate 6 (zero silent fallbacks: "NEVER catch an infrastructure error and return a synthetic "OPTIMAL", "APPROVED", or "deterministic_*" fallback object") Status: DESIGN — design document for remediation

1. Problem Statement

Multiple documented cases exist where infrastructure failures were caught and synthetic values returned instead of propagating the error:

LaunchDarkly console errors: The SPA through the Keycloak/oauth2-proxy gate at :3052 produced 11 console errors because the bundled Novu web unconditionally calls withLDProvider({clientSideID: window._env_.REACT_APP_LAUNCH_DARKLY_CLIENT_SIDE_ID || ""}), and with no LD key configured, it still initializes LD and fires network requests to app.launchdarkly.com/events.launchdarkly.com. The fix patches the bundle call site to a passthrough identity HOC that renders children without initializing LD when the client ID is falsy.

environmentId: environmentId || null in kutuma-api: The post-login token issued via getSignedToken(user, command.newOrganizationId, member) carries no environmentId (always null). Every environment-scoped endpoint validates environmentId from the JWT → 422. The fix patches getSignedToken to fall back to member.environmentId and backfills members.environmentId in MongoDB.

catch -> mock/default patterns: Throughout the codebase, infrastructure service calls may catch errors and return synthetic success, violating the fail-closed mandate.

The root enabler is that JavaScript try/catch blocks around infrastructure calls can silently return synthetic values instead of propagating errors.

2. Design Principles

2.1 Fail-Closed by Default

Every infrastructure service call must follow this pattern:

// ANTI-PATTERN (violates mandates):
try {
  const result = await someInfrastructureCall();
  return result; // success — fine
} catch (error) {
  // VIOLATION: returning synthetic success
  return { status: "OPTIMAL", data: null }; // WRONG
  // OR: returning { approved: true, data: null }; // WRONG
  // OR: returning { deterministic_: true, data: null }; // WRONG
}

// CORRECT PATTERN:
try {
  const result = await someInfrastructureCall();
  return result; // success — fine
} catch (error) {
  // ALWAYS fail closed — never return synthetic success
  throw error; // or return a typed error that the caller must handle
  // OR: return HTTP 502/503/422 as appropriate to the caller
}

2.2 Error Propagation Contract

All infrastructure call functions must follow one of two patterns:

Pattern A: Throw the original error (recommended for upstream handling)

async function callSurrealDB(query) {
  try {
    const result = await surreal.query(query);
    return result;
  } catch (error) {
    // Preserve the original error — don't wrap or synthesize
    throw error; // re-throw with original message/status
  }
}

Pattern B: Return typed error status (for API routes that must not throw)

async function callSolverGateway(payload) {
  try {
    const result = await nats.request('solver-gateway:v1.decisions', payload, { timeout: 5000 });
    return result;
  } catch (error) {
    // Fail closed with appropriate HTTP status
    if (error.status === 503) {
      return { error: 'SOLVER_UNAVAILABLE', hint: 'try later', status: 503 };
    }
    if (error.status === 400) {
      return { error: 'INVALID_PAYLOAD', hint: 'check request', status: 400 };
    }
    // Default: generic service unavailable
    return { error: 'SERVICE_UNAVAILABLE', hint: 'try later', status: 502 };
  }
}

CRITICAL: Pattern B must NOT return { status: "OPTIMAL" } or { approved: true } or any synthetic success. The hint field may provide guidance, but the status field must be an HTTP error status (400, 422, 502, 503) or the error must be thrown for upstream handling.

2.3 No Synthetic Success in Catch Blocks

Absolute prohibition: In any catch block, the following are FORBIDDEN:

Returning { status: "OPTIMAL" } or similar synthetic success markers
Returning { approved: true } or { granted: true }
Returning { deterministic_: true } or { deterministic: true }
Returning any object with a result field that contains synthetic data
Returning undefined or null as a "graceful" fallback
Returning {} (empty object) as "no data"
Merging synthetic values with real error info: { error: "msg", status: 503, synthetic: "OPTIMAL" }

Enforced behavior: Catch blocks must either:

Re-throw the original error: throw error
Wrap with context but preserve error status: throw new Error(${context}: ${error.message}) (preserves the error type)
Return HTTP error status only (for API route handlers, not utility functions): return new Response(JSON.stringify({ error: error.message }), { status: error.status || 502 })

2.4 API Route Fail-Closed Gates

Every API route must have an explicit fail-closed gate. The middleware pipeline should enforce this.

Route guard middleware pattern:

// Middleware that ensures fail-closed behavior
async function failClosedGate(req, res, next) {
  try {
    // Execute the route handler
    const result = await routeHandler(req, res);
    // If we get here, the handler did not throw —
    // but we must verify no synthetic success was returned
    if (result && result.status === "OPTIMAL") {
      // This should never happen — log and force 502
      return res.status(502).json({ error: "Invented success detected, fail-closed violated" });
    }
    if (result && result.approved === true && !result._real_infrastructure_result) {
      return res.status(502).json({ error: "Synthetic approval detected, fail-closed violated" });
    }
    res.json(result);
  } catch (error) {
    // Infrastructure error — never synthesize a success
    const status = error.status || 502;
    const errorBody = error.isSyntheticSuccess 
      ? { error: "Fail-closed violation: synthetic success would be returned" }
      : { error: error.message };
    return res.status(status).json(errorBody);
  }
}

NO: The middleware must never, in the catch block, return { status: "OPTIMAL", data: result } or similar.

2.5 Lint Rule for Catch Blocks

Add an ESLint rule (or equivalent lint for the language used) that flags any catch block that:

Returns a value (other than throw error)
Contains object literals with status: "OPTIMAL" or approved: true or deterministic: true
Contains Array.from() or fake generator patterns
Sets synthetic coordinates, suppliers, or organization IDs

The lint rule should fire at build time (or at least at lint time) for any catch block that does not throw error or return new Response(..., { status: 502 }).

2.6 Specific Fixes for Documented Cases

Case 1: LaunchDarkly bundle patch (already fixed per AGENTS.md)

The bundle was patched via entrypoint.sh sed patterns to replace user-facing Novu strings with Kutuma while keeping internal identifiers (Se.Novu, NovuProvider, @novu/*) intact. The LaunchDarkly fix patches the call site to a passthrough identity HOC when the client ID is falsy. This is a frontend fix, but the principle applies: never initialize an external service with empty/invalid config and return synthetic success.

Case 2: kutuma-api environmentId fix (already fixed per AGENTS.md)

The getSignedToken function in community.auth.service.js was patched so environmentId falls back to member.environmentId. The MongoDB was backfilled with each org's first environment ID. The kutuma-api container was recreated from the patched image kutuma-api:patched. This fixes the root cause but the design principle remains: infrastructure errors (missing environmentId) must fail closed, not synthesize a default.

Case 3: General catch-block audit

Audit all catch blocks in the codebase using a script:

# Search for catch blocks that return values (not throw)
rg "catch\s*\s*\{" C:\jo\apps\ --no-heading -n | grep -v "throw error" | grep -v "re-throw" | grep -v "throw.*error"

Any catch block not ending in throw error or properly returning an HTTP error must be fixed.

3. Implementation Plan

3.1 Step 1: Catch-Block Audit

Run a automated audit of all catch blocks in the codebase:

# Bash script to find catch blocks that might synthesize success
find C:\jo\apps -name "*.js" -o -name "*.ts" | xargs grep -l "catch" | while read file; do
  # Check if catch block returns a value (anti-pattern)
  if grep -q "catch" "$file" && ! grep -A 5 "catch" "$file" | grep -q "throw error"; then
    echo "POTENTIAL ISSUE: $file"
  fi
done

3.2 Step 2: Fix Each Catch Block

For each catch block identified:

If it throw error — leave as-is (correct)
If it returns a synthetic success — change to throw error or return HTTP 502/422
If it returns null/ undefined as "graceful" fallback — change to throw error
If it returns { status: "OPTIMAL" } — change to throw error or return 503

3.3 Step 3: Add Lint Rule

Add ESLint rule no-synthetic-success-in-catch that:

Flags any catch block body that is not just throw err or return new Response(..., { status })
Reports the line number and file
Runs in CI so the build fails if any such catch block exists

3.4 Step 4: CI Enforcement

Add the lint rule to the CI pipeline. Any PR with a catch block that doesn't throw error or return an HTTP error will fail CI. No exceptions.

3.5 Step 5: Document the Pattern

Update the AGENTS.md mandates section with the concrete pattern:

"All infrastructure service calls must be wrapped in try/catch that either throws the original error or returns an HTTP error status (400/422/502/503). Never catch and return a synthetic success object."
Include the anti-pattern examples and the correct pattern.

4. Verification

After remediation, verify:

[ ] No catch block in the codebase returns { status: "OPTIMAL" } or { approved: true } or { deterministic: true }
[ ] No catch block returns null, undefined, or {} as a "graceful fallback"
[ ] All catch blocks either throw error or return an HTTP error response with appropriate status
[ ] CI lint rule no-synthetic-success-in-catch passes
[ ] The three documented cases (LaunchDarkly, kutuma-api environmentId, general catch audit) are all fixed
[ ] A test script scripts/verify-fail-closed.mjs passes, which:
Starts an infrastructure service that deliberately fails
Makes a request that goes through the catch block
Asserts the response is an HTTP error (4xx/5xx), NOT a synthetic success
Asserts no status: "OPTIMAL" or approved: true in the response

5. Dependencies

This design depends on:

AGENTS.md mandates 3 and 6 (fail-closed, zero silent fallbacks)
Existing error handling patterns in the codebase
The LAUNCHDARKLY and kutuma-api fixes that are already applied
ESLint (or equivalent linting tool) configured in the project
CI pipeline that runs lint rules