Observability

Logging

Format

The backend emits structured JSON logs to stdout:

{"time": "2025-02-15 14:23:11,456", "level": "INFO", "name": "behavry.proxy.engine", "message": "OPA decision for agent=a1b2c3d4 tool=read_file action=read resource=/home/projects/report.pdf → allow (filesystem.read)"}

Log Levels by Logger

Logger	Level	What's Logged
behavry	DEBUG	All application logs
behavry.proxy.engine	INFO	Every OPA decision (agent, tool, action, resource, result)
behavry.policy.opa_client	INFO	OPA request/response timing
behavry.monitor.service	INFO	Anomaly detections, baseline updates
behavry.audit.service	WARNING (on error)	Audit write failures
uvicorn	INFO	HTTP request logs
sqlalchemy.engine	WARNING	Only SQL errors (not queries)

Key Log Events to Watch

Message Pattern	Meaning
OPA decision ... → deny	Tool call blocked
OPA decision ... → escalate	Tool call queued for human review
DLP findings for agent=...	Sensitive data detected in tool inputs
Baseline cache updated for agent	Agent baseline approved/refreshed
Prompt drift detected for agent	System prompt hash mismatch
Failed to write audit event	Audit pipeline error (DB issue)
Base policy push failed	OPA unreachable at startup
Behavry ready	App started successfully

Enabling Debug Logging

BEHAVRY_DEBUG=true

This sets DEBUG level on the root logger. Warning: very verbose — includes all SQL queries and HTTP request bodies.

---

SSE Live Feed

The audit stream (GET /api/v1/audit/stream) is the real-time observability surface for the dashboard. Event types:

event_type	Payload
tool_call	tool call allowed through
policy_deny	tool call denied
policy_escalate	tool call held for escalation
escalation	escalation created
alert	behavioral alert raised
system_notification	system message (e.g., restart countdown)

Keep-alive comments (:keepalive) are sent every BEHAVRY_SSE_KEEPALIVE_SECONDS (default: 15s) to prevent proxy timeouts.

---

Metrics

Behavry exposes OpenTelemetry metrics in Prometheus text format at GET /metrics (mounted at startup, Sprint O). The endpoint is unauthenticated by default in dev; set BEHAVRY_METRICS_TOKEN to require Authorization: Bearer <token> (pass it in your Prometheus scrape config).

A representative selection of the instruments emitted:

Instrument	Type	Notes
proxy_requests	Counter	Proxy requests (carries a tool label)
proxy_errors	Counter	Proxy-pipeline errors
policy_decisions	Counter	Policy decisions taken
opa_decide_errors / opa_errors	Counter	OPA evaluation / client errors
opa_decide_retries / opa_push_policy_retries	Counter	OPA retry counts
dlp_blocks	Counter	Requests blocked by DLP
rate_limit_blocked	Counter	Requests blocked by rate limiting
audit_write_errors	Counter	Failed audit writes
escalation_pending	Gauge	Pending escalations
active_sessions / registered_agents	Gauge	Live session / agent counts
db_pool_size / db_pool_checkedout / db_pool_overflow / db_pool_timeouts	Gauge / Counter	SQLAlchemy pool health
alert_loop_escalation_queue_depth / alert_loop_stale_tasks	Gauge	Alert-loop health
opa_decide_duration_seconds	Histogram	OPA decision latency
proxy_request_duration_seconds	Histogram	End-to-end proxy latency
policy_eval_duration_seconds	Histogram	Policy evaluation latency
db_query_duration_seconds	Histogram	DB query latency

Export to an OTLP backend

Metrics also export over OTLP with no code changes — set OTEL_EXPORTER_OTLP_ENDPOINT (and OTEL_EXPORTER_OTLP_HEADERS for auth). Examples:

# Grafana Cloud
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64 instanceid:apikey>

# Datadog
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.datadoghq.com/api/intake/otlp

You can also derive operational metrics directly from the audit log via SQL (examples below).

---

Key Queries for Operational Monitoring

Denial rate in last hour

SELECT
    COUNT(*) FILTER (WHERE policy_result = 'deny') AS denials,
    COUNT(*) AS total,
    ROUND(100.0 * COUNT(*) FILTER (WHERE policy_result = 'deny') / NULLIF(COUNT(*), 0), 1) AS deny_pct
FROM audit_events
WHERE timestamp > NOW() - INTERVAL '1 hour';

Top denied agents

SELECT agent_id, COUNT(*) AS deny_count
FROM audit_events
WHERE policy_result = 'deny'
  AND timestamp > NOW() - INTERVAL '24 hours'
GROUP BY agent_id
ORDER BY deny_count DESC
LIMIT 10;

OPA decision latency distribution

SELECT
    percentile_cont(0.5) WITHIN GROUP (ORDER BY latency_ms) AS p50,
    percentile_cont(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95,
    percentile_cont(0.99) WITHIN GROUP (ORDER BY latency_ms) AS p99
FROM audit_events
WHERE timestamp > NOW() - INTERVAL '1 hour'
  AND policy_result IS NOT NULL;

Escalations older than 20 minutes (approaching timeout)

SELECT id, agent_id, tool_name, target, timestamp, timeout_at
FROM escalations
WHERE status = 'pending'
  AND timeout_at < NOW() + INTERVAL '10 minutes'
ORDER BY timeout_at ASC;

DLP findings by severity (last 24h)

SELECT
    finding->>'severity' AS severity,
    COUNT(*) AS occurrences
FROM audit_events,
     jsonb_array_elements(dlp_findings) AS finding
WHERE dlp_findings IS NOT NULL
  AND timestamp > NOW() - INTERVAL '24 hours'
GROUP BY finding->>'severity'
ORDER BY occurrences DESC;

---

TimescaleDB Performance

Chunk Status

-- Show chunk sizes and compression state
SELECT chunk_name,
       pg_size_pretty(before_compression_total_bytes) AS before,
       pg_size_pretty(after_compression_total_bytes) AS after,
       compression_status
FROM chunk_compression_stats('audit_events')
ORDER BY range_start DESC;

Compression Policy

-- Enable compression (recommended for production)
ALTER TABLE audit_events SET (timescaledb.compress, timescaledb.compress_segmentby = 'agent_id');
SELECT add_compression_policy('audit_events', INTERVAL '7 days');

-- Check policy jobs
SELECT * FROM timescaledb_information.jobs WHERE application_name LIKE '%compress%';

---

Alerting Recommendations (Production)

Configure these alerts in your monitoring system:

Alert	Condition	Severity
OPA unreachable	OPA health check failing	Critical
High denial rate	> 20% denials in 5 min	Warning
Audit write failures	Any Failed to write audit event logs	Critical
DB connection pool exhausted	Pool wait time > 1s	Warning
Pending escalations	Any escalation pending > 20 min	Warning
No audit events	0 events in 5 min during business hours	Warning
Disk space	DB volume > 80% capacity	Warning

---

Distributed Tracing (Not Yet Implemented)

OpenTelemetry metrics are exported (see above), but distributed tracing (per-request spans) is not yet implemented. When added, each proxy request will carry a trace_id through:

• JWT validation
• Session check
• DLP scan
• OPA evaluation
• Backend forwarding
• Audit write

This will enable end-to-end latency attribution per enforcement stage.