Alerts & Escalations

Alerts

Alerts are generated by the Behavioral Monitor when anomalies or policy violations are detected. They live in the alerts table and appear in the Alerts page of the dashboard.

Alert Types

Every alert carries an alert_type and a severity, and is published on the event bus as an ALERT event. The types below are grouped by the subsystem that raises them.

Volumetric / statistical anomalies (Behavioral Monitor, z-score and rate based)

alert_type	Triggered by
frequency_anomaly	Request frequency deviates from the per-agent baseline
rate_spike	Request rate exceeded ~3× the rolling average
error_spike	Error rate spiked above baseline
resource_anomaly	Resource-usage metric outside baseline
volume_anomaly	Payload volume outside baseline

Integrity & drift

alert_type	Triggered by
system_prompt_drift	initialize prompt hash differs from the registered baseline (high → critical)
behavioral_drift_detected	Agent risk score on a sustained upward trend (medium)
injection_conditioning_suspected	Repeated injection findings within a window (high)

Injection & content

alert_type	Triggered by
inbound_injection_detected	Injection pattern detected in a response the agent received (high)
cross_session_dlp	Sensitive data correlated across an agent's sessions
DLP_FRAGMENT_REASSEMBLY	Fragmented sensitive data reassembled across multiple calls

Requester identity & trust reset (AOC-3 / AOC-4)

alert_type	Triggered by
requester_identity_mismatch	Requester identity on a call doesn't match the expected/declared requester
delegation_scope_probe	Delegation token requesting scopes beyond its grant (probing)
behavior_reversal	Agent allowed an action it had previously blocked for the same requester
requester_session_cycling	Same requester opens multiple sessions probing the same tool

Workflow baseline & blast radius

alert_type	Triggered by
workflow_depth_spike	Delegation depth exceeds the workflow baseline
workflow_tool_distribution_anomaly	Tool-usage distribution diverged from the workflow baseline
workflow_duration_anomaly	Session duration far outside the workflow baseline
workflow_participant_unexpected	An unexpected agent joined a workflow
blast_radius	A blast-radius limit (bulk/recipient/protected-path) was hit

Severity Levels

Severity	Meaning	Webhook delivery
info	Informational; no action required	Never
warning	Investigate; not immediately harmful	If BEHAVRY_WEBHOOK_MIN_SEVERITY=low or medium
critical	Immediate attention required	Yes (default threshold)

Alert Lifecycle

open → acknowledged → resolved

• open: Alert created, no human action taken
• acknowledged: Admin has seen it (PATCH /api/v1/alerts/{id} with {"status": "acknowledged"})
• resolved: Incident is closed ({"status": "resolved", "resolved_by": "admin"})

Example Alert Record

{
  "id": "b2c3d4e5-0000-0000-0000-000000000001",
  "agent_id": "a1b2c3d4-...",
  "alert_type": "system_prompt_drift",
  "severity": "critical",
  "timestamp": "2025-02-15T14:23:11Z",
  "status": "open",
  "details": {
    "provided_hash": "sha256:abc123...",
    "registered_hash": "sha256:def456...",
    "session_id": "s-..."
  }
}

---

Escalations

Escalations implement Human-in-the-Loop (HITL) approval for tool calls that OPA decides need human review before proceeding.

How Escalations Work

1. OPA returns {"result": "escalate", "reason": "...", "policy": "..."}
2. The proxy engine:
   • Writes an audit event with policy_result: "escalate"
   • Creates an Escalation record in the database
   • Registers an asyncio.Future keyed to the escalation ID
   • Holds the agent's HTTP connection open while waiting
3. An SSE event is published → the dashboard badge updates immediately
4. Admin reviews the escalation on the Escalations page
5. Admin takes action → the Future is resolved → the proxy continues

Agent Experience During Escalation

The agent's POST /mcp/v1 call blocks (long-poll). If approved, it receives the backend response. If denied or timed out, it receives:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32004,
    "message": "Escalation timed out — action auto-denied",
    "data": {"resolution": "timed_out"}
  }
}

Timeouts by Risk Tier

Risk Tier	Timeout
critical	5 minutes
high	15 minutes
medium	30 minutes
low	30 minutes
unknown	15 minutes

After timeout, a background task auto-denies the escalation. The agent receives an error and the human is no longer needed.

Resolution Options

Action	Result	Audit
Approve	Tool call forwarded to backend; agent receives result	policy_result: allow, policy_reason: "Escalation approved"
Deny	Error returned to agent immediately	policy_result: deny
Approve + Exception	Approved + creates a PolicyException for future identical calls (extendable up to 4 times)	policy_result: allow + exception created
Timeout	Background task auto-denies	policy_result: deny, policy_reason: "Escalation timed out — action auto-denied"

Escalation Record

{
  "id": "esc-uuid-...",
  "agent_id": "a1b2c3d4-...",
  "session_id": "sess-uuid-...",
  "tool_name": "delete_file",
  "action": "delete",
  "target": "/home/projects/sensitive_report.pdf",
  "mcp_server": "filesystem",
  "policy_reason": "File deletion requires human approval",
  "risk_tier": "medium",
  "status": "pending",
  "timeout_at": "2025-02-15T14:53:11Z",
  "request_payload": {
    "method": "tools/call",
    "params": {"name": "delete_file", "arguments": {"path": "/home/projects/..."}}
  },
  "timestamp": "2025-02-15T14:23:11Z"
}

API Operations

# List pending escalations
GET /api/v1/escalations?status=pending

# Approve
POST /api/v1/escalations/{id}/approve
Content-Type: application/json
{"notes": "Confirmed with team lead"}

# Deny
POST /api/v1/escalations/{id}/deny
Content-Type: application/json
{"notes": "Not authorized at this time"}

# Approve and create exception
POST /api/v1/escalations/{id}/approve
Content-Type: application/json
{"notes": "Pre-approved for temp cleanup", "create_exception": true, "exception_expires_at": "2025-12-31T00:00:00Z"}

---

Webhook Delivery

High-severity alerts trigger outbound webhooks to configured targets (Slack, SIEM, etc.).

JSON payload:

{
  "event": "alert",
  "alert_type": "system_prompt_drift",
  "severity": "critical",
  "agent_id": "a1b2c3d4-...",
  "timestamp": "2025-02-15T14:23:11Z",
  "details": {...},
  "link": "https://app.behavry.com/alerts"
}

Headers:

Content-Type: application/json
X-Behavry-Signature: sha256=<hmac-hex>

The signature is HMAC-SHA256 of the raw request body, using BEHAVRY_WEBHOOK_SECRET.

CEF format (for Splunk/ArcSight):

CEF:0|Behavry|AgentGateway|1.0|system_prompt_drift|System prompt drift detected|8|agent=a1b2c3d4 severity=critical

---

Behavioral Monitor Integration

The monitor subscribes to the event bus and updates per-agent rolling baselines (mean + stddev per metric). Anomaly detection runs on each event:

• Computes z-score: z = (value - mean) / stddev
• If |z| > 3.0 and enough samples exist (>= 10), creates an anomaly alert
• Metrics tracked: actions per minute, deny rate, escalation rate, tool diversity, error rate

Baselines are flushed to the database every 60 seconds by a background task.

See Observability for what metrics to watch.