Temporal Worker (pay-crons)

The pay-crons worker is the payment platform’s scheduled-job runtime. It is a long-running container that polls a single Temporal task queue (pay-crons), executes eight workflows on a cron cadence, and performs all side-effecting work (Hyperswitch REST calls, Prisma writes, Datadog events) inside activities.

It exists because of the Hyperswitch pivot: once Hyperswitch became the source of truth for transfers, settlements, disputes, and authorizations, the only jobs left for MVS to own are catch-up and MVS-specific monitoring that Hyperswitch does not natively model. The package (@mvs/pay-temporal-workflows) replaced the prior Finix-direct @mvs/payfac-jobs sync surface (8 payfac*Workflow definitions) with the current set.

Package layout

The source lives under packages/temporal-workflows/src/. The directory layout is load-bearing: tsup emits one entry per package.json exports subpath so the dist/ tree mirrors src/, because @temporalio/worker re-bundles dist/workflows/ at runtime through its own (webpack-based) sandbox bundler and needs it as a real on-disk directory.

Architecture at a glance

Two processes share the same image:

• The worker (dist/worker/start.js) — long-running, polls the queue, runs workflow + activity code. It never creates schedules.
• The register-schedules CLI (dist/cli/register-schedules.js) — a one-shot job that creates/updates the Temporal Schedules which enqueue the workflows. Without it the worker would poll an empty queue forever.

Worker entrypoint

src/worker/start.ts is intentionally thin. startPayCronsWorker():

1. Injects secrets via @mvs/secrets loadAppSecretsForRuntime("mvs-pay", env) — but only when isInfisicalConfigured() is true. In Kubernetes the ExternalSecret already projects the env, so this is usually a no-op there; it matters for local/dev boot. The scope is deliberately "mvs-pay" (shared with apps/mvs-pay and apps/mvs-pay-extensions) rather than introducing a new downstream identifier into the @mvs/secrets app union.
2. Builds a NativeConnection to TEMPORAL_ADDRESS (default localhost:7233). mTLS is enabled iff both TEMPORAL_TLS_CERT_PEM and TEMPORAL_TLS_KEY_PEM are present.
3. Creates the Worker on PAY_CRONS_TASK_QUEUE (default pay-crons) in TEMPORAL_NAMESPACE (default default), with workflowsPath resolved to dist/workflows/ and the union of all three activity modules registered.

1
const worker = await Worker.create({
2
  connection,
3
  namespace,
4
  taskQueue: TASK_QUEUE,
5
  workflowsPath: path.resolve(__dirname, "../workflows"),
6
  activities: {
7
    ...mvsPayActivities,
8
    ...webhookDlqDrainerActivities,
9
    ...successRateReconcileActivities,
10
  },
11
});

main() wires SIGINT/SIGTERM to worker.shutdown() (graceful drain), then await worker.run(). The Deployment sets terminationGracePeriodSeconds: 60 to let in-flight activities finish.

Environment variables

Resolved via @mvs/secrets at boot when Infisical is configured; otherwise read straight from the process env. See also the Environment Variables reference.

The eight workflows

Workflow code is deterministic — it only logs and forwards to an activity via proxyActivities. All I/O happens in activities. Each workflow returns a PayCronResult:

1
interface PayCronResult {
2
  processed: number;
3
  errors: number;
4
  details?: Record<string, unknown>; // free-form per-workflow counters
5
}

processed/errors are a deploy-grade heartbeat; details is the operator-facing bag visible in the Temporal Web UI. Activities are proxied at one of two timeout + retry tiers (all use startToCloseTimeout: "10m"):

1. Hyperswitch event replay (every 30 min)

runHyperswitchEventReplay lists Hyperswitch events created in the last lookbackHours (default 2h, sized for the 30-min cadence; limit default 500) via hyperswitch.events.list, skips delivered events, and calls hyperswitch.events.retry(event_id) for the rest. This is the catch-up safety net for the webhook delivery path: if Hyperswitch’s own outbox failed to deliver to mvs-pay-extensions, replay re-fires it.

details: { retriedCount, skippedCount }. “Delivered” is read from event.is_delivered or a status === "delivered" fallback.

2. Instant-payout settlement reconcile (every 15 min)

runInstantPayoutSettlement reads InstantPayoutRequest rows in REQUESTED status (default limit 100, oldest first), fetches each via hyperswitch.payouts.get(payoutId), and transitions the row:

• succeeded/success/completed → COMPLETED (+ completedAt)
• failed/cancelled/expired/reversed → FAILED (+ failedAt, failureCode, failureMessage)
• anything else (still processing) → left in place, counted as processed.

Rows with no hyperswitchPayoutId are skipped (nothing to reconcile). This is the poll-side complement to the PAYOUT_* webhook handler; see Payouts & Instant Payouts.

3. Residual estimation (monthly, 1st @ 06:00 UTC)

runResidualEstimation snapshots the last full calendar month per active MerchantExtension. For each merchant it pages hyperswitch.payments.list (merchant_id-scoped, pageSize 100, capped at 50 iterations) summing gross and connector fees, then writes a metadata.residuals[YYYY-MM] entry (grossAmount, connectorFees, netToMerchant, paymentCount, estimatedAt).

4. Residual drift checks (daily @ 10:00)

runResidualChecks re-aggregates the trailing lookbackHours (default 24h) per active merchant, computes an observed connector-fee rate in basis points (connectorFees / gross × 10000), and compares it to a per-merchant baseline at metadata.feeBaseline.expectedRateBps with a tolerance of metadata.feeBaseline.toleranceBps (else defaultDriftBps, default 50 bps = 0.5%). When the absolute deviation exceeds tolerance it opens a SYSTEM / HIGH Notification (href: /merchants).

Merchants with no expectedRateBps baseline configured are silently skipped — this job only alerts where someone has set an expected rate.

5. Dispute SLA monitor (daily @ 09:00)

runDisputeSlaMonitor finds DisputeExtension rows older than slaDays (default 5) where lastPlatformActionAt IS NULL, and opens a DISPUTE_DEADLINE / HIGH notification (href: /disputes). It de-dupes to one alert per dispute per 24h.

6. Device health monitor (every 30 min)

runDeviceHealthMonitor scans enabled Device rows (cap 1000) and, for any whose lastConnectedAt is older than offlineAfterMinutes (default 30, or never connected), opens a DEVICE_OFFLINE / HIGH notification (href: /devices), de-duped to one per device per 24h. See Devices & POS.

7. Webhook DLQ drainer (every 5 min)

runWebhookDlqDrainer (Phase 3 / M17.1) drains the inbound-webhook dead-letter queue. It selects WebhookDeliveryAttempt rows where status="pending" and lastAttemptAt is null or older than the 1-minute cool-off (default limit 100, oldest first), then re-runs the dispatch logic and updates the row.

Key behaviours:

• MAX_ATTEMPTS = 10. On the attempt that reaches 10, the row flips to status="failed" and emitDlqExhaustedEvent fires a Datadog event (alert_type: error, tags feature:webhook_dlq, service:mvs-pay-extensions).
• The dispatcher is replicated, not imported. dispatchHyperswitchWebhook is a deliberate copy of apps/mvs-pay-extensions/src/services/webhook-dispatcher.ts because workspaces can’t depend across the apps/ ↔ packages/ boundary (see Boundaries). It handles PAYMENT_INTENT_*, WEBHOOK_DISPUTE_*, and PAYOUT_* events; refund/mandate/unknown types are intentional no-ops that still mark the row processed. It applies the M17.2 ordering check (isAfterOrCloseEnough, 1s clock-skew tolerance) to avoid overwriting newer state with a stale replay.
• After draining, it re-counts remaining pending rows and emits the mvs_pay.webhook.dlq.depth gauge (M18.2) so the SLO monitor can fire on backlog growth.

details: { drained, exhausted, scanned }. The DB writes mirror only what the inbound handler does — no Hyperswitch API calls in this activity.

8. Success-rate reconcile (hourly)

runSuccessRateReconcile (Phase 4 / M23.5) drives the dynamic-routing health loop for Routing & Smart Retries. It iterates MerchantExtension rows with smartRetriesEnabled = true (and isActive), and for each with a hyperswitchProfileId calls the SDK client.routing.dynamic.getScoreWindow({ merchantId, profileId }). For every score-window entry it:

1. Emits the mvs_pay.smart_retry.success_rate gauge (tagged connector:*).
2. Tracks a per-(connector[:merchantConnectorId]) consecutiveLowCount in MerchantExtension.metadata.scoreWindowHistory.
3. When the score is below the threshold (LOW_SCORE_THRESHOLD = 0.5, overridable via the workflow threshold input) for SUSTAINED_RUNS_BEFORE_ALERT = 2 consecutive runs (= 2h at the hourly cadence), fires a warning Datadog event (feature:dynamic_routing).

The metadata write is done inside a $transaction that re-reads the row and merges only the scoreWindowHistory key, so it does not clobber a concurrent cron writing metadata.residuals (the residual estimator) or a dashboard config save.

Schedules (register-schedules)

src/schedules/pay-crons.ts is the single source of truth for cadence. It exports PAY_CRONS_SCHEDULES — a readonly PayCronScheduleSpec[] of { scheduleId, workflowType, cron, description }. The schedule IDs are the operator-facing handle in the Temporal UI:

src/cli/register-schedules.ts reads that array and, per spec, attempts client.schedule.create(...); on ScheduleAlreadyRunning it falls back to handle.update(...) so cadence edits in the source file land idempotently. Each spec is registered with:

• action: { type: "startWorkflow", workflowType, taskQueue }
• spec: { cronExpressions: [spec.cron] }
• policies: { overlap: "SKIP", catchupWindow: "1 hour" }
• memo: { description } (visible in the UI)

overlap: "SKIP" means a run is dropped if the prior run of the same schedule is still going — important for the longer monthly/daily aggregations.

$
# Idempotent — safe to run on every deploy.
$
pnpm --filter @mvs/pay-temporal-workflows start:schedules        # built (dist)
$
pnpm --filter @mvs/pay-temporal-workflows start:schedules:dev    # tsx (source)

The CLI logs a one-line created/updated/errored outcome per schedule and exits non-zero if any registration failed — this is what gates the ArgoCD PreSync hook.

Deployment

Manifests live in deploy/pay-crons-worker/ (Kustomize base + prod/staging overlays) and the ArgoCD Application in deploy/argo/pay-crons-worker-application.yaml. The worker runs in the payments Kubernetes namespace and is not PCI-scoped. See the broader Deployment Topology and GitOps & Kubernetes pages.

Image

A multi-stage Dockerfile (build context = repo root) mirrors the apps/mvs-pay-extensions image so the two payment images share an identical base / pnpm / Node / Prisma layout.

The tsup build inlines @mvs/* workspace packages (which ship raw TypeScript and can’t be loaded by plain node) via noExternal, while keeping @temporalio/* (native + sandbox-injected), @prisma/*, dd-trace, and zod external and resolved from the copied node_modules. dist/workflows/ is emitted as a real directory because the Temporal sandbox re-bundles it at runtime.

Deployment object

deploy/pay-crons-worker/base/deployment.yaml:

• 2 replicas, RollingUpdate (maxSurge: 1, maxUnavailable: 0). HA is for availability, not throughput — Temporal hands each task to exactly one worker.
• HPA 2..6 on CPU @ 60% (hpa.yaml). Workers are CPU-bound during deterministic replay; the ceiling is low because over-scaling adds connection churn without speeding bursty work.
• PDB minAvailable: 1, zone topology spread, and host anti-affinity.
• No Service. Outbound-only.
• Probes are pgrep -f 'worker/start.js' exec probes — the worker has no HTTP health surface. A documented future enhancement: write a heartbeat file on each poll cycle so liveness can detect a hung poll loop.
• Resources: requests 200m/384Mi, limits 750m/768Mi.
• securityContext: non-root, readOnlyRootFilesystem: true with a 64Mi /tmp emptyDir, all caps dropped, RuntimeDefault seccomp.
• Datadog APM via admission.datadoghq.com/enabled: "true" and downward-API DD_ENV/DD_VERSION from pod labels (the overlay replacements block copies the pinned image tag into the version label so DD_VERSION is the SHA).

ArgoCD + PreSync schedule-register Job

The ArgoCD Application (project: mvs-pay, source deploy/pay-crons-worker/overlays/prod, automated sync with prune + selfHeal, ServerSideApply, PruneLast) drives the rollout.

The critical piece is register-schedules-job.yaml, an Argo PreSync hook:

1
annotations:
2
  argocd.argoproj.io/hook: PreSync
3
  argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

The Job runs node dist/cli/register-schedules.js to completion before the Deployment rolls, on every sync, so schedules exist before (or in lockstep with) the new worker. It uses the same image, ServiceAccount, and secret bundle as the worker — only the command differs. backoffLimit: 3, activeDeadlineSeconds: 300, ttlSecondsAfterFinished: 600. If registration fails the hook fails and Argo halts the sync, leaving the prior worker running.

Secrets & network

externalsecret.yaml projects a pay-crons-worker-secrets Opaque Secret from Infisical at /apps/mvs-pay/pay-crons-worker/* (1h refresh, deletionPolicy: Retain), envFrom-mounted into both the worker and the Job. It carries the Temporal address/namespace/TLS pair, the Hyperswitch base URL + admin key, MVS_PAY_DATABASE_URL, the Infisical bootstrap quartet, and DD_API_KEY. See Secrets Management.

The NetworkPolicy allows ingress only from the monitoring namespace on 9090 (Prometheus) and egress to the in-cluster Hyperswitch router (:8080), Temporal/:7233, Postgres/:5432, Datadog+Infisical/:443, and ElastiCache Redis/:6379.

Boundaries

The worker is a separate runtime from the apps/mvs-pay-extensions Hono service and keeps its dependency graph deliberately narrow.

• @mvs/mvs-pay — workspace dep providing getMvsPayDb() + Prisma model types. Activities call it directly; workflows never import it (workflows are deterministic, DB calls go through activities). The worker’s only Prisma dependency is the generated @prisma/mvs-pay-client.
• No apps/mvs-pay-extensions import. The Hyperswitch HTTP client (src/lib/hyperswitch-client.ts) and the webhook dispatcher (in activities/webhook-dlq-drainer.ts) are deliberate standalone copies of the extensions versions — workspaces can’t depend across the apps/ ↔ packages/ boundary (see ADR-006 · Extensions as a separate app). These copies must be kept roughly in sync by hand for the methods the worker calls (events.list/retry, payments.list/get, payouts.get) and for the webhook event-type handling. There is no automated drift check.
• @mvs/mvs-pay-sdk — used only by the success-rate activity to reach the extensions service’s dynamic-routing route.
• @mvs/temporal — the planned cross-repo helper is not adopted; the worker constructs Temporal connections directly.

Metrics & observability

For day-to-day operations (replaying a stuck schedule, triaging DLQ exhaustion, on-call escalation), see the On-call runbook and the Smart Retries runbook.

See also