Problem Statement
The goal is to automate the creation of data quality checks for fintech lending and risk pipelines. The system must analyze schemas, business rules, historical patterns, and SLAs to produce a production-ready DQ test suite, alerting plan, runbook, and exception-handling policy. Outputs must align with regulatory-grade accuracy, deterministic test behavior, and low operational overhead.
Steps
Ingest Inputs
INGESTCollect table schemas, sample statistics, business rules, SLA requirements, and historical failure patterns from the data engineer. Parse and structure these inputs for downstream analysis.
Analyze Schemas and Data Characteristics
EXECUTEReview schemas, field types, nullability, reference keys, and distribution statistics. Identify fields requiring deterministic quality checks and reconciliation rules.
Generate Data Quality Test Suite
EXECUTEDraft a full DQ test suite including schema validation, null checks, distribution drift tests, and reconciliation checks using provided business rules and historical patterns.
Set Thresholds and Alerting Plan
EXECUTEDetermine baseline thresholds from historical data, define alert severity based on business impact, and document the alert routing logic and SLA expectations.
Validate and Document Runbook and Exceptions
EXECUTEValidate that the runbook includes actionable resolution steps for common failures and define an exception-handling workflow requiring explicit approval. Ensure all documentation is complete and production-ready.
Prompt Configurations
DQ Test Suite Generation
You are a rigorous, production-focused data quality copilot for fintech lending pipelines. Using the provided schemas, statistics, business rules, SLAs, and historical failure patterns, generate a complete Data Quality (DQ) test suite. Requirements: - Tests must be deterministic. - Include schema checks, null checks, distribution checks, reconciliation logic, and rule-based validations. - Thresholds must reference historical baselines only. - Do not invent fields or rules not present in the input. - Produce output in a structured, engineering-friendly format. - Maintain a rigorous, quality-focused, production-ready tone.
Alerting and Runbook Drafting
Act as a data quality reliability engineer. Based on the provided DQ tests, business impact tiers, and SLA requirements, generate: 1. A prioritized alerting plan with thresholds and routing. 2. A runbook containing step-by-step resolution paths for common failure scenarios. 3. An exception-handling workflow requiring explicit approval. Follow these constraints: - Do not fabricate metrics or thresholds; derive from given data. - Keep instructions operational and unambiguous. - Tone must be rigorous and production-ready.
Example Inputs/Outputs
Schemas for loan_applications and credit_scores; sample stats showing stable distributions; business rules requiring non-null applicant_id and reconciliation with credit bureau feed; SLA requiring alerts within 15 minutes; historical failures include null spikes in income field.
Generated DQ test suite including schema consistency checks, null constraint enforcement on applicant_id, distribution drift checks for income, and reconciliation tests with credit bureau records. Alert plan assigns high severity to applicant_id null spikes with 15-minute SLA. Runbook outlines steps to validate upstream ingestion and reprocess affected partitions. Exception workflow requires data lead approval with documented justification.