Problem Statement
This workflow supports on-call data infrastructure leads in handling incidents from triage to postmortem. It provides structured steps to assess alerts, determine blast radius, communicate with stakeholders, form root-cause hypotheses, and produce a blameless postmortem with action items. The goal is to ensure fast, calm, and systematic incident resolution with clear accountability and prevention measures.
Steps
Ingest and Parse Incident Inputs
INGESTCollect and parse alert text, sample error logs, DAG visualization, recent deployment history, and affected systems. Normalize these inputs into a structured set of incident signals for downstream steps.
Assess Impact and Determine Blast Radius
EXECUTEEvaluate which pipelines, data products, or stakeholders are affected. Prioritize triage actions based on severity, customer impact, and propagation risk.
Draft Stakeholder Communication
EXECUTEGenerate a concise, calm, and timely status update summarizing what is known, what is being investigated, and next update expectations.
Form Root-Cause Hypotheses
EXECUTEAnalyze logs, DAG failures, and deployment diffs to propose plausible root-cause hypotheses without assuming blame. Identify likely failure points or regressions.
Create Blameless Postmortem
EXECUTEProduce a structured postmortem documenting the timeline, contributing factors, validated root cause, action items with owners and due dates, and long-term prevention measures.
Prompt Configurations
Incident Triage and Communication Prompt
You are an on-call data infrastructure assistant. Your task is to process the provided incident inputs and return: (1) immediate triage steps in priority order based on blast radius, (2) a clear, calm, blameless stakeholder update message, (3) 2–5 plausible root-cause hypotheses grounded only in the supplied data, and (4) a complete blameless postmortem including timeline, contributing factors, confirmed root cause if known, action items with owners and due dates, and long-term prevention measures. Tone must be calm, systematic, and action-oriented. Do not invent systems or logs; rely only on given information. If information is missing, state assumptions explicitly.
Example Inputs/Outputs
Alert: DAG 'daily_user_metrics' failed for 3 consecutive runs. Logs: Timeout connecting to warehouse. DAG Visualization: Failure in extract → load step. Recent Deployments: New connection-pool settings deployed 2 hours ago. Affected Systems: Analytics dashboards using daily metrics.
Triage Steps: (1) Validate warehouse connectivity for all pipelines, (2) Roll back recent connection-pool configuration, (3) Re-run failing DAG tasks, (4) Notify analytics stakeholders. Stakeholder Update: 'We are investigating failures in the daily_user_metrics pipeline tied to warehouse timeouts. We are validating connectivity and reviewing the recent connection-pool deployment. Next update in 30 minutes.' Root-Cause Hypotheses: (1) Misconfigured connection-pool settings reducing available connections, (2) Intermittent warehouse network throttling, (3) DAG task retry exhaustion due to upstream slowness. Postmortem: Incident timeline reconstructed; root cause identified as overly aggressive connection-pool settings. Action items assigned to Infra Eng (rollback + validation by Friday) and Data Platform Lead (add connection saturation alerting by next sprint).