Problem Statement
The project aims to systematically identify, categorize, and assess the failure modes of an LLM agent based on its goals, tools, personas, and prior incidents. The workflow must produce actionable risk insights, including severity and likelihood ratings, detection signals, mitigations, and high‑risk test cases. This supports safer deployment, structured monitoring, and proactive risk reduction.
Steps
Ingest Agent Context
INGESTCollect and parse the agent's goals, tools, user personas, and known incidents. Normalize this information into a structured form suitable for downstream analysis.
Brainstorm Failure Modes
EXECUTEGenerate a categorized list of concrete, public-safe failure modes that the agent might encounter based on the provided context. Group failures into logical risk categories such as reasoning errors, tool misuse, or user-misinterpretation issues.
Rate Severity and Likelihood
EXECUTEAssign severity and likelihood scores to each identified failure mode using a consistent rubric. Prioritize high-risk items for deeper analysis.
Define Detection Signals
EXECUTEFor each failure mode, enumerate observable signals or telemetry patterns that would help detect the issue in real time or via monitoring pipelines.
Draft Mitigations and Guardrails
EXECUTERecommend mitigations, guardrails, and design adjustments for each failure mode. Ensure recommendations are constructive and public-safe.
Produce High-Risk Test Cases
EXECUTEFor failure modes rated as high-risk, create specific, non-sensitive test cases for evaluation and regression testing.
Prompt Configurations
Failure Mode Analysis Prompt
You are an AI risk analysis assistant. Given the agent's goals, tools, user personas, and known incidents, generate: (1) a categorized list of public-safe and concrete failure modes, (2) severity and likelihood ratings, (3) detection signals, (4) mitigations, and (5) high-risk test cases. Keep tone risk-aware and constructive. Avoid sensitive or harmful scenarios. Do not invent tools or personas beyond the provided details. Organize output clearly by category and maintain consistency across all sections.
Example Inputs/Outputs
Agent goal: Summarize internal project updates for employees. Tools: Document search, calendar lookup. User personas: project managers, engineers. Known incidents: occasional incorrect attribution of project owners.
Generated risk report including categories such as factual accuracy issues, tool misalignment, and context-misinterpretation risks. Severity/likelihood ratings assigned, with detection signals like unusually high uncertainty indicators or mismatched document metadata. Mitigations included stricter metadata validation and grounding checks. High-risk test cases focused on updates with ambiguous ownership and overlapping teams.