Observability

Precept ships with optional OpenTelemetry (OTel) instrumentation. Setting a single environment variable causes the server and Workflow worker to begin emitting distributed traces and structured logs to an OTLP-compatible collector. With the variable unset, the OTel SDK is loaded but no exporters are registered — the process produces no telemetry traffic and incurs no overhead from network attempts.

What's instrumented

Out of the box, the following are auto-instrumented and produce spans:

Layer	Coverage
Incoming HTTP requests	Every request reaching the Express server gets a root span with the route, method, status code, and timing.
Outgoing HTTP requests via Node http / https	Calls made through the built-in http/https modules (e.g. third-party SDKs that use them internally).
Outgoing HTTP requests via fetch	Node's native fetch (built on undici) is instrumented separately so SSO proxy, Anthropic, MCP, and ingestion adapter calls all produce client spans.
Express middleware	Each middleware in the request chain emits a span, so you can see where time was spent in routing, body parsing, auth, etc.
oRPC handlers	Each oRPC procedure call gets its own span nested under the request span.
PostgreSQL queries	Every pg query executed by the server or worker produces a span with the SQL text and parameter count.
pino log records	Application log lines are forwarded to the OTLP logs endpoint alongside traces.

Trace context is propagated automatically between layers, so a single incoming request shows up as a tree: HTTP request → middleware → handler → outgoing HTTP/DB calls.

Enabling OpenTelemetry

To turn instrumentation on, set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable on both the server and Workflow worker processes. Point it at the base URL of an OTLP/HTTP-compatible collector:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc.cluster.local:4318

When this variable is set:

• Traces are exported to <endpoint>/v1/traces
• Logs are exported to <endpoint>/v1/logs

A trailing slash on the endpoint is tolerated and normalized internally.

When this variable is unset, the OTel SDK starts but registers no exporters. No exports are attempted; no connection errors appear in logs.

Configuration reference

Configuration is primarily via standard OTel environment variables, plus one Precept-specific variable that controls database query redaction (see Database query redaction below). Set these on the same processes (server, worker) where they take effect.

Variable	Effect
OTEL_EXPORTER_OTLP_ENDPOINT	Base URL of the OTLP/HTTP collector. Unset → no exporters registered.
OTEL_SERVICE_NAME	Service name attached to all telemetry. Defaults to precept-server for the API server and precept-worker for the Workflow worker.
OTEL_RESOURCE_ATTRIBUTES	Comma-separated key=value pairs attached as resource attributes to every span and log. Use this for per-deployment tags such as tenant labels, environment, or deployment ID. Example: tenant_label=acme,environment=prod,deployment.id=42
OTEL_EXPORTER_OTLP_PROTOCOL	Wire protocol. Unset → traces and the in-process log processor use OTLP/HTTP JSON, but the pino log forwarder defaults to OTLP/HTTP Protobuf. Set to http/json to make every exporter use JSON, or http/protobuf for uniform Protobuf.
OTEL_LOG_LEVEL	Internal log level of the OTel SDK itself (e.g. debug). Useful for diagnosing exporter connectivity issues.
PRECEPT_OTEL_CAPTURE_DB_STATEMENT	Set to true to disable database query redaction and emit the full SQL text on db.statement attributes. See Database query redaction.

TIP

Most collectors accept both JSON and Protobuf, so leaving OTEL_EXPORTER_OTLP_PROTOCOL unset is fine. Set it explicitly only if you've constrained your collector to a single protocol.

Database query redaction

By default, the db.statement attribute on every database span is replaced with the literal string [redacted]. This protects against incidental leakage of business or user data into your traces — the database client renders parameter values into the SQL text before the query is sent, so without redaction every row of every insert, update, and lookup would appear in your trace storage.

Other database attributes (db.system, db.name, db.operation, query timing, status) are unaffected, so you still see which operation ran against which table and how long it took.

To temporarily disable redaction for debugging — for example, when diagnosing a slow query in a non-production environment — set:

PRECEPT_OTEL_CAPTURE_DB_STATEMENT=true

WARNING

Enable full db.statement capture only in environments where the trace data is appropriate to expose. The captured SQL will contain whatever your application wrote — including user IDs, email addresses, encrypted credential blobs, workflow definitions, and any other column data that flowed through the query.

Example: pointing at the OpenTelemetry Collector

The reference path is to run the OpenTelemetry Collector inside your cluster and forward from there to your eventual backend (Honeycomb, Grafana, Datadog, an on-prem store, etc.). A minimal Collector config that accepts OTLP and re-exports it is:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  # ...your downstream exporter here

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [/* your exporter */]
    logs:
      receivers: [otlp]
      exporters: [/* your exporter */]

Then set on Precept:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability.svc.cluster.local:4318"
  - name: OTEL_SERVICE_NAME
    value: "precept-server"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment=production"

If you're using the Helm chart, these are passed through normal env: overrides — see Helm Chart Values.

Example: pointing directly at a SaaS

Most observability SaaS vendors accept OTLP/HTTP directly and can substitute for the Collector. Consult your vendor's docs for the endpoint URL and the auth header they require, then set:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.eu.example-vendor.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20your-token-here

OTEL_EXPORTER_OTLP_HEADERS follows the standard OTel format: comma-separated key=value pairs, URL-encoded.

Verifying it's working

With OTEL_EXPORTER_OTLP_ENDPOINT set, hit any endpoint on the server and the corresponding span should appear in your collector or backend within a few seconds (spans are batched).

For local debugging, set OTEL_LOG_LEVEL=debug on the server process — the OTel SDK will log every export attempt to stderr, including retries when the collector is unreachable.

Disabling instrumentation

Unset OTEL_EXPORTER_OTLP_ENDPOINT (or remove it from your deployment manifest) and restart the affected pods. The OTel SDK will still load, but no exporters will be registered and no telemetry traffic will be produced.