Extensions Service

apps/mvs-pay-extensions is the internal payment-orchestration service. It is a standalone Hono HTTP server that wraps Hyperswitch’s REST API with MVS-specific behaviour — extension-table persistence, Plaid instant payouts, POS device management, custom billing, settlement/residual logic — and exposes a single typed surface that every consumer reaches through the @mvs/mvs-pay-sdk.

It runs in the payments Kubernetes namespace on EKS, not as Next.js API routes inside apps/mvs-pay. It is the only service in the platform that holds the Hyperswitch admin API key and talks to the Hyperswitch router, and it sits outside the PCI cardholder-data ring.

Why it exists as a separate service

Two ADRs pin down the two foundational choices. Read them in full; the summary here is deliberately short.

Separate EKS service, not Next API routes (ADR-006)

Hyperswitch orchestration could have lived in apps/mvs-pay’s Next.js API routes. It was extracted because:

• Clean PCI / non-PCI boundary. Extensions sits outside the PCI ring; only Hyperswitch (router + card-vault) is inside. Keeping orchestration in its own process keeps the boundary auditable.
• Cross-repo consumers. apps/mvs-pay is not the only caller — healthos and mvs-c2 call the extensions service directly over HTTP. tRPC-style in-process coupling does not work across repo boundaries.
• Independent scaling, observability, and deployment cadence. The payment hot path scales and ships separately from the merchant/operator UI.
• EKS operational features (mTLS, network policies) that Vercel functions don’t offer.

Hono, not tRPC (ADR-003)

The service is service-to-service infrastructure that needs typed routes, fast cold start, simple middleware, and easy request validation.

• Hono 4.12.23, with @hono/zod-validator for request schema validation and the ContentfulStatusCode literal-union pattern for c.json(body, status) typing.
• tRPC was rejected because the service is consumed by multiple repos over HTTP; tRPC’s tightly coupled types don’t map across repo boundaries. The typed @mvs/mvs-pay-sdk provides equivalent type safety over a stable HTTP contract.
• Express / Fastify were rejected as heavier with weaker TS ergonomics.

PCI scope and the Hyperswitch boundary

The only secret that lets this service mutate money movement is the Hyperswitch admin API key (HYPERSWITCH_ADMIN_API_KEY), loaded at runtime from Infisical. Connector credentials (e.g. Finix) live inside Hyperswitch’s merchant_connector_account configuration, managed through the Hyperswitch Control Center — never in this service’s environment. The .env.example contract is explicit about this:

$
# Hyperswitch (no Finix credentials here — Hyperswitch holds them in
>
# its merchant_connector_account configuration via the control-center).
$
HYPERSWITCH_API_BASE_URL=https://hyperswitch.payments.internal
$
HYPERSWITCH_ADMIN_API_KEY=
$
HYPERSWITCH_WEBHOOK_SECRET=

All outbound calls to Hyperswitch go through one typed client, src/services/hyperswitch-client.ts. It is JSON-in / JSON-out, reads the admin key from runtime config, retries on 5xx + network failures, and wraps each call in a circuit breaker (withCircuitBreaker from @mvs/cache, keyed hyperswitch:<METHOD>:<path>). Higher-level orchestration (combining a Hyperswitch call with extension-table writes) lives in the route handlers, not in the client.

Request flow

Everything is assembled in createHonoApp() in apps/mvs-pay-extensions/src/app.ts. The server entrypoint (src/server.ts, run as node dist/server.js) boots that app via @hono/node-server, listens on PORT (default 3004), and wires SIGINT / SIGTERM to a graceful shutdown that closes the HTTP server and the mvs-pay DB pool. Local dev uses tsx watch src/dev.ts.

On startup, if Infisical is configured (isInfisicalConfigured()), secrets are injected into process.env before config is parsed. If injection fails, the service logs and continues with whatever env vars are present — it does not hard-fail on Infisical.

The middleware chain

The order on the authenticated surface is fixed and load-bearing. From app.ts:

// Order matters: auth → rate-limit → idempotency → audit-log → handler.

Global middleware (app.use("*", …)) runs first for every request: CORS, then request logging (method, path, status, durationMs). The four chain stages below are mounted on the /v1 sub-app.

1. Auth — internal secret + tenant context

authMiddleware() enforces the service-to-service boundary only. Every consumer presents the rotated MVS_PAY_EXTENSIONS_INTERNAL_SECRET in the x-internal-secret header.

• If internalSecret is unset on the service, it returns 503 GATEWAY_NOT_CONFIGURED and refuses the request.
• The presented secret is compared in constant time (timingSafeEqual). A mismatch or missing header returns 401 UNAUTHORIZED.
• Tenant context is read from x-org-id / x-org-slug. If neither is present it returns 400 TENANT_REQUIRED.
• x-request-id is propagated (a synthetic req_… id is generated if absent), and { tenant, requestId } is set on the Hono context (GatewayContext in src/types.ts).

2. Rate limiting

rateLimit() is a Redis-backed sliding bucket via the @mvs/cache Redis client (INCR + EXPIRE on ratelimit:<scope>:<bucket>). The scope is the organizationId from tenant context; the bucket defaults to the request path.

• It is mounted per route group (both the collection path and the /* subtree), not globally — payments, refunds, payouts, disputes.
• Webhook ingest uses a shared global bucket (bucketKey: () => "global", allowMissingTenant: true) so a single misbehaving upstream cannot starve other tenants.
• Fail-open semantics: on a Redis outage the limiter logs and lets the request through — taking traffic beats dropping legitimate writes during a cache hiccup. Over-limit responses return 429 RATE_LIMITED with a Retry-After header.

Limits come from loadRateLimitConfig(), overridable per environment:

3. Idempotency

idempotency() makes repeated mutations safe to retry. It is mounted only on the POST routes the spec calls out: /payments, /payment-links, /refunds, /payouts, /instant-payouts, /customers, and /plaid/exchange.

Failure handling is conservative: a lookup error proceeds to the handler (better a possible duplicate than a failed legitimate write), and a concurrent-write unique violation (P2002) is treated as a benign no-op. Per spec § M17.3.

4. Audit log

auditLog() persists one AdminAuditLog row per mutation (POST / PATCH / PUT / DELETE); it short-circuits on reads. It is best-effort — a failure to persist the row is logged but never fails the request (Datadog watches for that log line).

What it records: a derived action (<method>.<routePath>), a best-effort (entityType, entityId) inferred from the /v1/<entity>/<id> path, the requesting tenant in metadata, the response status, the client IP (x-forwarded-for), and the user agent.

Because the service is service-to-service, it does not know the real end user; AdminAuditLog.userId is set to the synthetic system:mvs-pay-extensions, and the actual user identity is carried inside the consumer. Per spec § M16.3.

The /v1/* surface

The authenticated API is a child Hono app mounted at /v1. Every route group is defined in its own file under src/routes/ and exposed through @mvs/mvs-pay-sdk. The contract for each lives in the API Reference tab.

Unmatched routes return a structured 404 NOT_FOUND; unhandled exceptions are caught by app.onError and return 500 INTERNAL_ERROR (the underlying message is only echoed in development).

Webhook ingest — /webhooks/*

Webhook ingress is not authenticated by the internal secret — these are inbound calls from external systems. Each endpoint instead verifies a signature inside its handler and is protected by the shared global rate-limit bucket (allowMissingTenant: true).

Hyperswitch webhook handling

The main ingest at /webhooks/hyperswitch is the canonical path for lifecycle events (PAYMENT_INTENT_*, WEBHOOK_REFUND_*, WEBHOOK_DISPUTE_*, MANDATE_*, PAYOUT_* — the UPP-normalized event set).

1. Verify. Hyperswitch signs each payload with an HMAC of the raw body using the merchant’s payment_response_hash_key (exposed to this service as HYPERSWITCH_WEBHOOK_SECRET). It checks SHA-512 in x-webhook-signature-512 first, with a SHA-256 fallback in x-webhook-signature-256. Constant-time compare. A failure returns 401; an unset secret returns 503.
2. Persist before dispatch (M17.1). A WebhookDeliveryAttempt row is upserted, idempotent on hyperswitchEventId, so Hyperswitch redeliveries collapse onto one drain target. Already-processed events short-circuit with { ok: true, replay: true }.
3. Dispatch. dispatchHyperswitchWebhook() (in src/services/webhook-dispatcher.ts) fans the event out into MVS extension tables — e.g. confirming a TransferExtension, opening a DisputeExtension
   • Notification on WEBHOOK_DISPUTE_OPENED, updating SettlementExtension approval state on PAYOUT_*. It applies a webhook-ordering check (M17.2): events older than the local row’s updatedAt (minus a 1 s skew tolerance) are skipped as stale. Unmapped event types fall through silently — Hyperswitch’s own events table remains the source of truth.
4. Outcome. Success marks the row processed. A dispatch failure leaves it pending, increments attemptCount, records lastError, and returns 500 so Hyperswitch retries on its own backoff.

The dispatcher is a pure function reused in three places: inbound ingest, the admin DLQ retry, and the Temporal webhookDlqDrainerWorkflow that re-runs pending rows every 5 minutes (see the Temporal Worker).

Admin — DLQ replay (/admin/webhooks/dlq)

The dead-letter-queue admin surface (src/routes/admin/webhooks-dlq.ts, spec § M17.1) lets an operator inspect and manually retry failed webhook deliveries. It is tenant-scoped and sits behind the internal secret plus the audit-log middleware (auth → audit-log), but it is mounted outside /v1.

Retry behaviour: rows already processed return 409 ALREADY_PROCESSED; an unknown id returns 404; a successful retry marks the row processed and bumps attemptCount; a failed retry records lastError and returns 500 DISPATCH_FAILED. All rows are filtered by tenant.organizationId.

Health, readiness, and discovery

Five public, unauthenticated endpoints. None of them call Hyperswitch; they report only the service’s own liveness and whether the Hyperswitch connection is configured.

CORS

CORS is applied globally as the first middleware. The allowed origin is resolved per environment by resolveCorsOrigin():

• In development, origin is "*".
• In every other environment, origins come from MVS_PAY_EXTENSIONS_CORS_ORIGINS (comma-separated) or default to https://pay.mvscloud.com, https://healthos.mvscloud.com, https://c2.mvscloud.com.
• A "*" origin in a non-development environment throws at startup — a wildcard is not allowed in staging/production.

Allowed methods are GET, POST, PUT, PATCH, DELETE, OPTIONS; allowed headers include Authorization, x-internal-secret, x-org-id, x-org-slug, x-request-id, and idempotency-key; credentials: true.

Configuration

Config is parsed once by a Zod schema in src/config/index.ts and cached. Key inputs (full list in Environment Variables):

In production, secrets are sourced from Infisical at /apps/mvs-pay/mvs-pay-extensions/{dev,prod} — see Secrets Management. For how the service is built, shipped, and probed, see Deployment Topology and GitOps & Kubernetes.

For the new owner

Things to keep honest as you work:

• Auth here is a boundary, not a user check. Real user identity lives in the consumer. Don’t reach for userId in this service — it doesn’t have one (audit rows use the synthetic system:mvs-pay-extensions).
• Rate limiting is partial. Only payments / refunds / payouts / disputes and webhook ingest have limits. Adding a new money-moving route? Mount a rateLimit() for it deliberately.
• /readyz is a config probe, not a connectivity probe. It will report ready against a wrong-but-present admin key.
• The audit log and rate-limiter both fail open. That is intentional, but it means neither is a hard control — pair them with monitoring, not assumptions.