Problem Statement
The workflow helps product teams interpret A/B test results by assessing both statistical and practical significance. It standardizes decisions on whether to ship variant B, stay with A, continue testing, or redesign the experiment. The workflow ensures rigorous, consistent evaluation using predefined confidence and lift thresholds.
Steps
Ingest Test Inputs
INGESTCollect raw A/B test metrics including conversion rates, sample sizes, test duration, confidence level, and hypothesis details. Validate that all required fields are present and properly formatted.
Validate Test Setup
EXECUTECheck test duration, confidence level, sample adequacy, and hypothesis clarity. Flag potential issues such as novelty effects, premature termination, or mismatched sample sizes.
Calculate Statistical Significance
EXECUTECompute statistical significance using provided conversion rates and sample sizes. Determine whether the observed difference between A and B meets or exceeds the 95% confidence threshold.
Assess Practical Significance
EXECUTEEvaluate the effect size and determine whether the lift exceeds the 2% minimum meaningful threshold. Consider segment-level variations when assessing real-world impact.
Generate Recommendation
EXECUTEProduce a ship/no-ship/continue testing/redesign recommendation based on statistical significance, practical significance, test validity, and known risks. Include clear trade-offs and caveats.
Prompt Configurations
A/B Test Interpretation Prompt
You are an A/B test evaluation assistant. Using the provided A/B test metrics (conversion rates, sample sizes, confidence levels, duration, and hypothesis), provide a rigorous interpretation. Instructions: - Determine statistical significance without inventing numbers. - Compute and summarize practical significance (effect size and percentage lift). - Follow these rules for recommendations: * Recommend 'ship B' only if confidence >=95% AND lift >2%. * Flag novelty effects if test duration <7 days. * Consider segment differences when provided. - Include a concise, decision-focused explanation of trade-offs and risks. - Tone must be rigorous, balanced, and decision-focused. Return four sections: (1) statistical significance, (2) practical significance, (3) recommendation, (4) risks/caveats. Only use information explicitly provided in the input.
Example Inputs/Outputs
Hypothesis: Variant B will increase checkout conversion. Duration: 5 days A: 12.0% conversion (n=10,000) B: 13.0% conversion (n=10,200) Confidence level: 95%
Statistical Significance: The difference between A (12.0%) and B (13.0%) is statistically significant at the 95% confidence level. Practical Significance: The lift is 1.0 percentage point (~8.3% relative), exceeding materiality in relative terms but below the required 2% absolute lift threshold. Recommendation: Continue testing. Lift is promising but does not meet the >2% absolute lift requirement for shipping. Duration is under 7 days, so novelty effects are possible. Risks/Caveats: Novelty effects, short duration, and absence of segment-level analysis may obscure true performance. Consider extending test length before making a final decision.