Troubleshooting observability
Why deliveryHealth is unknown, runs are incomplete or conflicted, and definitions show no runs: truthful states, not silent bugs.
Most “broken” observability UI states are accurate reports of partial telemetry. Start from the state vocabulary, then check host flush and identity emission.
Why is deliveryHealth: unknown?
This is the default. The local server only promotes healthy when it can prove a clean terminal delivery: ended run, causal ordering, zero gaps, no trace-alias conflict, and no ingest rejections. Still-running work, incomplete reconciliation, and partial ordering stay unknown.
UI tooltip: The server has not observed enough delivery evidence to call this run healthy or degraded.
What to check
- Did the run actually end (
run:endingested)? - Is the host wrapper flushing before freeze/exit? See Runtime setup.
- Any
degradedbadge instead? Then some records were rejected: inspect local diagnostics / rejected counts, not client invention of healthy.
unknown is not “healthy but unlabeled.” Never treat missing proof as success.
Why is my run incomplete?
UI copy: Telemetry ended without a run:end record: the run may still be executing out of view, or its process exited before reporting a terminal status. (Plus gap wording when sequence gaps exist.)
Common causes
- Serverless / Worker / Convex action returned without a bounded flush
- Process killed mid-generation (OOM, deploy, hard timeout)
- Transport failures after lifecycle start records already landed
- Operation deadline expired while terminal records never arrived (presentation may mark incomplete without rewriting raw records)
What to do
- Bind the correct host wrapper for your runtime
- Confirm
ObservabilityFlushResult.status === 'drained'viaonDrain - For Convex, use first-party
action/internalAction/createCruxConvexso flush is automatic - Re-run with
crux dev+ tunnel when debugging cloud workers
Why is my run conflicted?
- A trace alias identifies more than one logical run… when
traceAliasConflictis set - Otherwise stored terminal evidence … conflicts with what was previously recorded
Usually a propagation / identity bug: two logical runs sharing an alias, or contradictory terminal records under one runId. Fix identity at the source; do not merge client-side.
Why doesn’t my definition show any runs?
Catalog zero-activity copy: No runs have referenced this definition yet.
Checklist
| Check | Detail |
|---|---|
| Did it run under a Crux adapter / primitive that emits spans? | Static-only kinds never get direct activity. |
| Does the handle carry a canonical id? | Anonymous prompts/contexts/tools and non-literal composition ids omit DefinitionRef. Compositions need required authored id. |
| Is the kind a contributor / structural child? | You may see “referenced by N runs” rather than being the subject of a run. |
| Eval-owned kinds | Primary lens is Eval; secondary direct runtime only when the manifest declares it (e.g. scorer / scoring.judge). |
| File-loaded skills | Registry skills join; bare file-loaded directory ids are a different id space and intentionally do not emit refs. |
| Since-deleted definitions | Runs still filter by historical id. Imported exact manifests resolve the old definition; a missing manifest stays explicitly unresolved. |
View Runs appears only when the activity rollup reports runCount > 0. An empty state with no button is intentional.
Why is Run Detail missing Catalog links?
Links come from runtime definitionRefs on the run/span. If refs were never emitted (anonymous handle, non-instrumented path), there is nothing to link. Identified runs resolve only against their exact imported deployment manifest. A missing manifest is manifest-unresolved; a present manifest without that id is definition-unresolved. The current Catalog is a separately labeled comparison, never a fallback.
Why did stream completion miss usage / finish reason?
First-party adapters now capture completion metadata on stream settle and surface completion failures as CruxAdapterError. If you still see empty finish/usage:
- Confirm you are on the first-party adapter packages (not a custom dialect that skips the completion hook)
- Confirm the provider actually returned usage (OpenAI streams need usage enabled; Crux requests it for OpenAI streams)
- Distinguish aborted / error finishes from missing metadata
Progressive tool-call argument fragments are never exposed: wait for completed tool calls on the stream completion.
Why don’t my OTel spans parent correctly?
withTelemetry() activates a real SDK span around instrumented work. Requirements:
- Register a
TracerProviderbefore installing the plugin (standard path), or pass a lightweightexporter - Do not expect one OTel span object to survive
run:suspend; resume starts a new root span sharing the CruxtraceId - Propagate across queues with
injectCruxPropagationCarrier/extractCruxPropagationCarrier(W3Ctraceparent/tracestate)
See Telemetry.