Problem Statement
Data teams need a structured way to determine which data quality checks matter most so that alerts are meaningful and not noisy. The goal is to evaluate critical datasets, identify failure modes, define appropriate checks, and assign alert ownership while ensuring a gradual rollout and handling known false positives. The workflow should produce a coverage matrix, recommended checks, threshold rationale, routing, and noise‑reduction guidelines.
Steps
Ingest Inputs and Parse Dataset Context
INGESTCollect and parse the provided list of critical datasets, SLAs, consumer dashboards, recent incidents, and existing checks. Normalize the inputs into a consistent structure that can be evaluated for coverage gaps and criticality.
Rank Dataset Criticality
EXECUTEEvaluate each dataset using SLAs, usage patterns, dashboard dependencies, and past incidents. Produce a ranked list indicating which datasets require the most urgent or robust monitoring.
Map Failure Modes and Define Recommended Checks
EXECUTEIdentify potential failure modes for each high‑criticality dataset and map them to appropriate check types such as freshness, volume, schema, or distribution. Produce a dataset-to-check coverage matrix and recommend additional checks if gaps exist.
Tune Thresholds and Handle Known False Positives
EXECUTEGenerate threshold rationales based on SLAs, historical patterns, or operational constraints. Explicitly incorporate known sources of false positives and propose guardrails that reduce noise without suppressing real incidents.
Assign Ownership, Routing, and Rollout Plan
EXECUTEDetermine which team or individual owns each dataset and the corresponding alerts. Outline routing rules and propose a gradual rollout plan that starts with high‑critical datasets and expands coverage over time.
Prompt Configurations
Monitoring Design Prompt
You are a data reliability analyst. Using the provided datasets, SLAs, dashboards, incidents, and current checks, generate a full monitoring design. Follow these requirements: - Rank dataset criticality based on SLAs, dependencies, and incident history. - Map failure modes and identify recommended checks (freshness, volume, schema, distribution). - Produce a dataset-to-check coverage matrix. - Recommend threshold rationales that avoid noise. Do not invent numeric thresholds; only infer patterns logically. - Identify known false positives and propose mitigation approaches. - Define alert routing and ownership using only provided or inferable roles. - Include a gradual rollout plan. Tone: practical, reliability-focused. Do not invent tools or vendor-specific features. Use only the information given.
Example Inputs/Outputs
Critical datasets: orders_daily, customers_master, payments_raw. SLAs: orders_daily <2h latency, payments_raw <1h latency. Dashboards: revenue_daily uses orders_daily and payments_raw. Incidents: orders_daily late twice last month. Current checks: freshness on orders_daily only.
Criticality ranking: 1) orders_daily (high dependency, SLA breaches), 2) payments_raw (feeds same dashboard), 3) customers_master (medium). Coverage matrix: orders_daily → freshness, volume, schema; payments_raw → freshness, volume; customers_master → schema. Recommended checks: add distribution checks for orders_daily and payments_raw. Add freshness check for customers_master. Threshold rationale: freshness should align with SLAs; volume thresholds should tolerate known weekend dips. Alert routing: data platform owns orders_daily and payments_raw; CRM team owns customers_master. Noise reduction: suppress volume alerts for orders_daily on weekends due to predictable low volume. Rollout: start with orders_daily enhancements, then payments_raw, then customers_master.