Observability

Observability in Cessy starts from the event-sourced flow. When behavior is wrong, follow the path from command to event to projection to policy or integration.

What to observe

Activity first

Use Activity to trace runtime behavior before inspecting low-level logs. It should show the business sequence and platform side effects in one place.

For durable policy recovery, a started handler without a completion receipt has an unknown effect, not proof that nothing happened. Check canonical effects before an audited reconciliation decision. A sealed plan is eligible for a receipt-derived not-applied retry only while all children remain pending, as after an atomic command rollback. Completed effects and recoverable publication receipts retain their own recovery paths; do not replay the source command.

Workspace snapshot recovery reads the current active lifecycle epoch before constructing policy workers. It does not replace the active configuration or reactivate a retired Workspace. Missing, expired, or mismatched scope authority remains blocked and requires investigation of that existing work.

For a batch item’s command execution reference, call get_execution with the same app and Workspace. When AutomationLog evidence is absent, the tool reads the indexed canonical command event receipt under the exact tenant/source. Complete receipts return effectState: committed, event IDs and global positions without business payloads. They prove the command append, not downstream policy completion or external effects; inspect the returned event references with inspect_policy_followups_for_event. Unavailable, inconsistent, incomplete, or oversized receipts (more than 256 events) are inconclusive. A missing receipt or log never authorizes replay, including for commands that emit no events.

Delayed command dispatch

For delayed commands, get_policy_status reports payload-free delayedQueue counts: pending (including pending recovery), overdue, and failed. Overdue or failed timers make the policy DEGRADED; future pending timers make it BACKLOGGED. Successful materialization is not successful dispatch or provider acceptance. Inspect the Cron dispatch worker and exact queue disposition before recovery; do not reissue the source domain command. Production reads inspect existing owned collections under the status deadline and never provision a queue.

Inspect retained Policy workers

Operators can use the repository’s scripts/inspect-policy-workers.ts dry run to classify a bounded window of retained Workspace workers as required, ambiguous, or an orphaned generation. It reads immutable worker ownership and existing runtime-scope, TestRun and recovery metadata. Missing ownership, unfinished release, held effects and incomplete inventory retain the worker. Historical published workers have no retirement authority from this initial inspector.

The command requires a read-only database connection and a private output path. Its stdout contains aggregate counts and duration; exact container identities stay in the private report. A separately confirmed operator command can consume only a complete private report, reclassifies each candidate under the lifecycle row fence, checks exact Docker labels again, and preserves all volumes. Follow the repository’s docs/operations/policy-worker-lifecycle.md for budgets, confirmation and proof-review rules. Container age, a completed test or improved resource headroom alone does not authorize retirement. Design MCP authoring requires no additional tools or steps.

Verify a restored database

A successful SQL restore is only the first recovery check. Restore the declared owners, roles and grants separately when the archive omits them, then use the matching engine revision’s ordinary migration and runtime schema checks. The qualified PostgreSQL 16.9 restore path recognizes specific equivalent catalog representations while preserving rejection of unknown schema and authority changes. Do not reset migration fingerprints to silence an admission failure.

Verify the saved Published configuration, ownership epochs and outstanding work through an isolated read-only inspection before enabling policies, triggers or other workers. Unknown external outcomes still need reconciliation. The synthetic engine regression does not establish recovery of a particular production archive, files, credentials or a single-writer cutover.

Verify Workspace resource delivery

Treat declared resources, inspected container limits and a temporary operator override as separate evidence. Replacement reapplies the declared template; it must preserve workspace files and must not silently retain a temporary override. Historical Workspace events can predate the desired-resource snapshot.

The resource-event rollout requires its exact source compatibility proof and the completed Production App, worker and E2E gates before publishing new Agent resource declarations. A merge or an accepted rollout flag is insufficient. Inherited catalog expansion and worker changes require their separate reviewed Delivery operations. Changes to any reviewed reader or worker dependency require a new compatibility review before delivery. Keep failed deployment receipts and link the qualified successor separately. During the transition, do not rebuild projections using the old engine; the new engine can reconstruct resource snapshots from the retained events.

Diagnose a slow page

In SigNoz, open Cessy Production Request Causality and set the exact UTC window around the slow navigation. Start with Slow route requests, then open the matching row to inspect its trace. Routes are normalized; for example, an execution detail is reported as /a/[appId]/design/activity/executions/[id], never with the real app or execution identifier.

The trace separates application phases from database.pool.acquire waits and database.query execution. Query spans expose a governed queryName, finite query family and operation, pool/service class, outcome, and duration. They do not expose raw SQL, bind values, or record identifiers. Browser Web Vitals use a finite frontendRoute and can share the same trace as the server navigation; the trace reference is not a metric label.

Long database-acquisition spans can also include delayed Web-process callbacks. Compare the same process and UTC window with event-loop/CPU evidence before attributing them to PostgreSQL saturation. Legacy Workspace cache reads copy large graphs cooperatively while preserving independent caller-owned values; their scheduling qualification does not establish zero database hydration cost.

Missing route samples, parent links, query names, or browser/server joins are unknown evidence, not proof that latency was zero. Use the PostgreSQL dashboard for aggregate database fingerprints and pivot by the same UTC window, service/release, governed query name, operation/family, duration, and call rate.

Verify worker trace coverage

In SigNoz, correlate naturally occurring worker attempts using cessy-background-worker, the worker profile, release and an absolute UTC window. Worker readiness, scraped queue metrics and lifecycle logs do not prove that worker traces reached the backend. Isolated workers preserve the explicitly configured OTLP export controls and release identity while keeping their existing credential boundary. If worker traces are missing, inspect the actual Node process configuration; Docker’s declared environment or its init process can still contain settings removed before Node starts. Keep missing causal evidence open until a retained worker trace demonstrates the expected relationship.

Error log

Use error logs when Activity points to a platform, policy, adapter, webhook, auth, or projection failure. Preserve correlation ids when escalating.

Adapter failures and non-success HTTP responses are also retained as bounded, redacted diagnostics on the policy execution, including when policy code handles the response and completes successfully. Use get_execution for the run; when an Error Log entry exists, use get_error_context. Each normalized entry can record operation identity, substitution/login/call phase, safe method/path, status, attempts, optional elapsedMs, retryDecision, refreshDecision, httpResponseReached, transportCategory, an optional allowlisted transportCode, runtime source, and an opaque runtimeFingerprint. A receipt retains at most 16 chronological entries and preserves terminal evidence.

httpResponseReached: true means that at least response headers arrived; false identifies a pre-HTTP failure, while older records can render unknown. The finite transportCategory values are dns_resolution, tcp_refused, tcp_reset, tcp_timeout, tls_handshake, tls_certificate, egress_denied, upstream_unreachable, aborted, and unknown_transport. Only a finite allowlist of symbolic low-level values is rendered as transportCode; unrecognized codes are omitted. Unknown ERR_TLS_* and ERR_SSL_* families can still classify as tls_handshake without retaining the raw code. Arbitrary messages, nested causes, credential values, auth headers, cookies, hostnames, network addresses, query strings, and full upstream URLs are not part of the safe receipt.

For a controlled reproduction, test_adapter accepts exactly production, workspace:<workspaceId>, or legacy * as sourceScope and can call only a declared, parameterless GET or HEAD operation. Workspace ownership is checked before authentication or adapter I/O. Production and Workspace tests report sourceBinding: runtime; * reports sourceBinding: global_credentials_only and is not an exact runtime test. Results include httpResponseReached, elapsed time, terminal attempts/retry and refresh evidence when present, source binding, and runtimeFingerprint. At result level, httpResponseReached is true when any login or safe-probe leg reached response headers, even if a later retry failed before its response.

Custom-code policy worker_not_deployed diagnostics are kept aligned with the published model after publish: Cessy resolves stale open diagnostics when the policy was removed, no longer requires a worker, or now has a deployed worker.

For custom-code policies, targetCommands is the finite upper bound of command types the handler may return, not a requirement that every invocation return one. An intentional null, empty-array, or void result on a conditional settle/retry path is recorded as a successful no-command execution; it is not an execution error. Add safe handler logs or metrics for attempt and settle counts when operators need more detail than the built-in success/no-command result; never log projection rows or credentials.

Before explicitly enabling durable outbound webhooks, alert on reconciliation backlog, expired claims, oldest outstanding age, dead letters, and repeated lease loss. These aggregate signals are scoped by Cell, tenant, and effective app and exclude endpoint, headers, payload, work id, and consumer name. An ambiguous webhook outcome should be reconciled against the receiver before it is retried.

Durable policy planning failures retain a bounded diagnosis for the invocation, separate from child-command failures. The phase distinguishes authorization, planner preparation, plan construction, and plan persistence; the retained cause uses a finite class/code and a scrubbed signature rather than raw source, messages, stacks, or credentials. If Error Log capture fails, the internal recovery-job receipt still retains the safe diagnosis. A completed recovery job may contain a reconciled policy failure. Missing child evidence or a failed response does not authorize replay, because authorization or plan persistence may already have committed.

Policy execution diagnostics

Use Design MCP list_policy_executions for a bounded AutomationLog page. It defaults to a frozen 24-hour window, accepts an explicit window of at most 31 days, returns at most 100 compact records, and enforces a database deadline of at most two seconds. Its opaque cursor binds the effective Production or Workspace app, filters, time window, and the (loggedAt, executionId) boundary. Start a fresh traversal to see inserts that arrived after the first page.

Treat the response’s availability and completeness fields as part of the result. A deadline, cancellation, capacity rejection, or query failure is reported as unavailable and is not evidence of absence. A next cursor or output truncation means the page is incomplete. Projection evidence remains the page authority. With a correlation or causation filter, bounded canonical evidence is shown only as a projection-lag probe; without that filter, lag is not measured and an empty page does not prove that no canonical execution exists. get_engine_health uses the same bounded last-hour path and reports policy state as UNKNOWN when execution evidence is unavailable or truncated.

The index-safe path reads flat scalar AutomationLog fields. Historical scalar records remain supported, but legacy array-valued equality through generic JSON containment is not used. Operators should complete the timestamp/scalar audit and optional backfill in the repository runbook at docs/operations/policy-execution-diagnostics.md before rollout.

For work that has not reached a terminal AutomationLog row, use Design MCP get_policy_status. Its Durable Policy Work section reports queued, claimed, executing, waiting_for_retry, or reconciliation_required, with the work and correlation IDs, exact config/runtime revisions, attempt budget, lease epoch and deadline, last safe boundary, and next recovery eligibility. Payloads, command intents, credentials, and worker identities are excluded.

• Operator action: wait means the current lease or automatic recovery path still owns the work.
• Operator action: reconcile means either an ambiguous effect or exhausted attempt-zero progress needs explicit review.
• Do not reissue the domain command while durable work is outstanding. That creates new work; it does not safely retry the existing invocation.

For projection-ready work still at attempt zero, status also identifies the minute recovery producer, its last successful observation, the next disposition boundary, and any five-minute progress breach. The first healthy cycle refreshes the exact durable wake-up. Subsequent minutes retain a bounded continuation while a peer in the exact tenant/App/config/build has a live lease or a new completion. That evidence never resets the original due age. Without progress, or after the five-minute limit, still-unstarted work becomes explicit reconciliation work with recovery_progress_exhausted. This needs operator attention and is not autonomous completion. The reported disposition boundary is the next progress evaluation, not a promised execution time. Old binary revisions receive this disposition only after their own producer observed the work; unobserved old-revision work stays unchanged for its build, a compatibility grant, or operator review, and is never executed under new code. Failed queue admission does not count as a healthy cycle. Existing customer effects are never replayed for verification. Operator-approved recovery still requires the exact runtime or a scoped compatibility grant. For missing recovery admission, verify that the runtime worker process received the exact grant through its scrubbed entrypoint, not just that the container configuration contains it. Recovery grants and native-batch recovery/capacity controls are passed only to the runtime profile and retain their normal validation. One exclusive recovery scope owner attempts each explicitly granted work ID through target-only claims within its existing deadline. Ordinary work retains its fairness slice. This preserves queue serialization, the five-minute progress bound and refusal to execute ungranted work; queue capacity is still required.

Historical policy projection callbacks after a config promotion use the sealed snapshot only when its tenant, App, source, expiry and exact engine implementation are valid. The requested projection must retain its canonical name, collection and structural behavior in the active configuration. Reads retain the active scoped store and collection lifecycle fences; the historical manifest never replaces the active runtime. Missing or incompatible evidence remains an explicit revision mismatch and cannot authorize a retry of ambiguous effects. This applies only to projection reads; adapter and file callback authority is unchanged.

Background-worker migration signals

Policy compensation Shadow jobs compare pending recovery intent with its runtime scope and report whether the sources match, advanced, are missing, or could not be read authoritatively. A completed Shadow receipt verifies only that comparison; it does not mean compensation executed or that Cron errors have been resolved. Use the canonical policy lifecycle to verify the actual recovery outcome.

In SigNoz, admission pilot metrics use jobName=internal.pilot. Filter by worker profile to distinguish capability pools; traces retain the exact pilot job name. Compensation Shadow producer and observer metrics keep separate job names.

Agent-specific traces

For repeated runtime-worker restarts, preserve the exact UTC error and release identity. A V8 allocation fatal can occur while Docker later reports healthy and OOMKilled=false; these observations describe different moments and failure mechanisms. Compare the effective Node heap, container RAM limit, and retained runtime allocations over repeated work and configuration revisions.

The runtime worker defaults to one top-level claim per replica. Recovery releases its owned snapshot runtime’s platform listeners after work settles. Increasing RAM requires measured heap/native headroom and bounded retention; it does not replace lifecycle cleanup. Apply any profile change through protected Delivery and verify the actual worker limits and original durable work identity. Outstanding or unknown effects must still be reconciled before retrying a source command.

For runtime agents, use Design MCP list_agent_runs while a wake may still be active, then get_agent_run, get_agent_session_history, get_agent_run_tool_calls, and get_agent_run_tool_surface after completion. For wake_agent policies, get_policy_status also shows the Wake Queue state so operators can distinguish pending, retrying, succeeded, and dead-lettered wake intents. When a policy wake contains a Codex runtime runId, list_agent_runs and get_agent_run reconcile stale non-terminal rows with CodexAgentRunBoard completion state. For builder agents, also inspect Design MCP tool calls, workspace changes, validation output, and promotion records.

A completed wake policy can mean that its durable wake intent was accepted; check the Wake Queue and the requested agent run for actual execution. Isolated runtimes deliver accepted intents to the runtime worker, with periodic reconciliation for lost delivery. An expired attempt with an uncertain effect requires reconciliation; repeating the original command or creating another wake is not evidence of recovery. Fallback and stale-recovery entrypoints also preserve unknown effects: recovery requires an explicit unattempted-claim marker and unchanged current lease evidence. Attempt zero or an invalid/expired timestamp alone never makes replay safe.

Debugging sequence

1. Confirm target tenant, app, and environment.
2. Confirm the published model contains the expected command or projection.
3. Execute or replay the smallest failing path.
4. Inspect Activity.
5. Check error logs and affected integrations.
6. Fix the model, permission, projection, policy, or integration.
7. Re-run the smoke test.

get_policy_status also returns its existing report in structuredContent.data, including appId, policyName, policyType, wakeQueue, and durablePolicy. A pending or processing wake intent proves acceptance, not completed agent work. A succeeded wake intent proves delivery of the wake; use agent-run evidence for completion of the agent task. Source command or projection success alone does not prove downstream policy execution.