Problem Statement
The goal is to take a list of data engineering tasks and transform it into a production-ready orchestration design. This includes defining a clear DAG blueprint, mapping explicit dependencies, and incorporating reliability mechanisms such as retries, backfills, logging, metrics, and cost controls. The workflow should help a Data Engineering Lead produce consistent, idempotent, and well-documented orchestration designs.
Steps
Ingest Task Specifications
INGESTCollect the provided task list including descriptions, dependencies, frequency, failure history, and cost constraints. Normalize fields and ensure each task has a unique identifier and idempotency statement.
Analyze Dependencies and Execution Behavior
EXECUTEReview each task for required upstream data sources, execution timing, and resource needs. Identify any missing or conflicting dependency information and synthesize a complete dependency map.
Generate DAG Blueprint and Reliability Policies
EXECUTEProduce a structured DAG design with explicit node relationships, scheduling, and execution constraints. Define retries with exponential backoff, backfill strategy, idempotency checks, and cost-aware execution patterns.
Create Observability Checklist
EXECUTEDraft logging requirements, required metrics (duration, cost, failures), and actionable alert rules. Ensure each job captures context-rich logs and that alert thresholds reflect production reliability standards.
Produce Operational Runbook
EXECUTEDocument operational procedures including on-call steps, manual backfill instructions, common failure remediation, and escalation paths. Ensure all instructions are clear, repeatable, and suitable for production use.
Validate Output Against Constraints
EXECUTECheck for adherence to idempotency, explicit dependencies, retry rules, logging expectations, metric coverage, backfill rules, and cost control mechanisms. Flag inconsistencies or missing requirements.
Prompt Configurations
DAG Design Generation Prompt
You are an engineering-focused orchestration consultant. Using the provided task list with descriptions, dependencies, execution frequency, failure history, and cost constraints, generate the following: 1. A complete DAG blueprint with explicit dependencies. 2. Retry and backfill policies (retries must use exponential backoff; backfills must support date ranges). 3. An observability checklist including logging, metrics (duration, cost, failure counts), and actionable alert rules. 4. A production-ready operational runbook. 5. Cost control mechanisms for expensive or high-frequency tasks. Constraints: - Do not invent tasks; only use the provided inputs. - Ensure every task is idempotent and dependencies are explicit. - Logging must include contextual metadata. - Maintain a professional, engineering-oriented tone. Output should be structured, concise, and production-ready.
Example Inputs/Outputs
Tasks: load_raw_sales (runs hourly, depends on no upstream, low failure rate, moderate cost); transform_sales (runs hourly, depends on load_raw_sales, high failure rate, high cost); publish_kpis (runs hourly, depends on transform_sales, low cost, stable). Constraints: enforce idempotency, retry with exponential backoff, add logging and metrics.
DAG Blueprint: load_raw_sales -> transform_sales -> publish_kpis. Retry Policy: load_raw_sales (2 retries, exponential backoff), transform_sales (4 retries, exponential backoff due to high failure history), publish_kpis (2 retries). Backfill Policy: support hourly ranges, ensure upstream completion before downstream. Observability Checklist: structured logs including run_id, timestamps, input/output counts; metrics for duration, cost per run, failure counts; alerts on repeated transform_sales failures. Runbook: steps for rerunning failures, instructions for triggering hourly backfills, escalation guidelines. Cost Controls: transform_sales monitored for cost spikes and flagged if exceeding 20% of monthly budget allocation.