Alerts & Escalations

Alerts

Alerts are generated by the Behavioral Monitor when anomalies or policy violations are detected. They live in the alerts table and appear in the Alerts page of the dashboard.

Alert Types

alert_type	Triggered By	Default Severity
anomaly	Metric deviates > 3σ from behavioral baseline	warning
system_prompt_drift	initialize provides a prompt hash different from baseline	high → mapped to critical in alerts
baseline_drift	Tool call uses a tool not in the registered manifest	critical
rate_limit	Agent exceeds configured action rate (not yet implemented)	warning
dlp_violation	DLP scanner finds sensitive data in tool inputs	critical
policy_deny_spike	Unusual number of denials in a short window (not yet implemented)	warning

Severity Levels

Severity	Meaning	Webhook delivery
info	Informational; no action required	Never
warning	Investigate; not immediately harmful	If BEHAVRY_WEBHOOK_MIN_SEVERITY=low or medium
critical	Immediate attention required	Yes (default threshold)

Alert Lifecycle

open → acknowledged → resolved

• open: Alert created, no human action taken
• acknowledged: Admin has seen it (PATCH /api/v1/alerts/{id} with {"status": "acknowledged"})
• resolved: Incident is closed ({"status": "resolved", "resolved_by": "admin"})

Example Alert Record

{
  "id": "b2c3d4e5-0000-0000-0000-000000000001",
  "agent_id": "a1b2c3d4-...",
  "alert_type": "system_prompt_drift",
  "severity": "critical",
  "timestamp": "2025-02-15T14:23:11Z",
  "status": "open",
  "details": {
    "provided_hash": "sha256:abc123...",
    "registered_hash": "sha256:def456...",
    "session_id": "s-..."
  }
}

---

Escalations

Escalations implement Human-in-the-Loop (HITL) approval for tool calls that OPA decides need human review before proceeding.

How Escalations Work

1. OPA returns {"result": "escalate", "reason": "...", "policy": "..."}
2. The proxy engine:
   • Writes an audit event with policy_result: "escalate"
   • Creates an Escalation record in the database
   • Registers an asyncio.Future keyed to the escalation ID
   • Holds the agent's HTTP connection open while waiting
3. An SSE event is published → the dashboard badge updates immediately
4. Admin reviews the escalation on the Escalations page
5. Admin takes action → the Future is resolved → the proxy continues

Agent Experience During Escalation

The agent's POST /mcp/v1/{server_id} call blocks (long-poll). If approved, it receives the backend response. If denied or timed out, it receives:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32004,
    "message": "Escalation timed out — action auto-denied",
    "data": {"resolution": "timed_out"}
  }
}

Timeouts by Risk Tier

Risk Tier	Timeout
critical	5 minutes
high	15 minutes
medium	30 minutes
low	30 minutes
unknown	15 minutes

After timeout, a background task auto-denies the escalation. The agent receives an error and the human is no longer needed.

Resolution Options

Action	Result	Audit
Approve	Tool call forwarded to backend; agent receives result	policy_result: allow, policy_reason: "Escalation approved"
Deny	Error returned to agent immediately	policy_result: deny
Approve + Exception	Approved + creates a PolicyException for future identical calls (extendable up to 4 times)	policy_result: allow + exception created
Timeout	Background task auto-denies	policy_result: deny, policy_reason: "Escalation timed out — action auto-denied"

Escalation Record

{
  "id": "esc-uuid-...",
  "agent_id": "a1b2c3d4-...",
  "session_id": "sess-uuid-...",
  "tool_name": "delete_file",
  "action": "delete",
  "target": "/home/projects/sensitive_report.pdf",
  "mcp_server": "filesystem",
  "policy_reason": "File deletion requires human approval",
  "risk_tier": "medium",
  "status": "pending",
  "timeout_at": "2025-02-15T14:53:11Z",
  "request_payload": {
    "method": "tools/call",
    "params": {"name": "delete_file", "arguments": {"path": "/home/projects/..."}}
  },
  "timestamp": "2025-02-15T14:23:11Z"
}

API Operations

# List pending escalations
GET /api/v1/escalations?status=pending

# Approve
POST /api/v1/escalations/{id}/approve
Content-Type: application/json
{"notes": "Confirmed with team lead"}

# Deny
POST /api/v1/escalations/{id}/deny
Content-Type: application/json
{"notes": "Not authorized at this time"}

# Approve and create exception
POST /api/v1/escalations/{id}/approve
Content-Type: application/json
{"notes": "Pre-approved for temp cleanup", "create_exception": true, "exception_expires_at": "2025-12-31T00:00:00Z"}

---

Webhook Delivery

High-severity alerts trigger outbound webhooks to configured targets (Slack, SIEM, etc.).

JSON payload:

{
  "event": "alert",
  "alert_type": "system_prompt_drift",
  "severity": "critical",
  "agent_id": "a1b2c3d4-...",
  "timestamp": "2025-02-15T14:23:11Z",
  "details": {...},
  "link": "https://app.behavry.com/alerts"
}

Headers:

Content-Type: application/json
X-Behavry-Signature: sha256=<hmac-hex>

The signature is HMAC-SHA256 of the raw request body, using BEHAVRY_WEBHOOK_SECRET.

CEF format (for Splunk/ArcSight):

CEF:0|Behavry|AgentGateway|1.0|system_prompt_drift|System prompt drift detected|8|agent=a1b2c3d4 severity=critical

---

Behavioral Monitor Integration

The monitor subscribes to the event bus and updates per-agent rolling baselines (mean + stddev per metric). Anomaly detection runs on each event:

• Computes z-score: z = (value - mean) / stddev
• If |z| > 3.0 and enough samples exist (>= 10), creates an anomaly alert
• Metrics tracked: actions per minute, deny rate, escalation rate, tool diversity, error rate

Baselines are flushed to the database every 60 seconds by a background task.

See Observability for what metrics to watch.