![]() ![]() With AGILE we address the core reason why many practitioners are tempted to and every so often do apply statistics in sub-optimal or outright wrong ways: the fact that the classical statistics, used in most of A/B testing, are not fit to the use-case and thus are both very inefficient and prone to misuse.įor example, classical procedures for calculating statistical significance require fixing the sample size (e.g. Why Is There a Need for a New Statistical Method in A/B Testing? ![]() Using these flexible early stopping rules results in an average gain starting from ~20% in the worst case scenario (true difference is just below the minimum effect of interest) and reaching ~80% in the best case where the true lift of the tested variant is significantly higher or lower than the minimum effect of interest. The efficiency gains compared to fixed sample tests come from the use of rules for early stopping for efficacy and for futility (lack of practically-significant effect, non-superiority). Having a reliable measure of the effects of any AB testing treatment and the uncertainty of the measure enables decision-makers to make smart choices and move their business in the right direction with greater confidence. The result from applying the AGILE approach to A/B testing is a greatly improved efficiency and thus ROI (Return on Investment) while maintaining the ability to control the uncertainty (error probability) in the experiments. Post-test estimates for % true lift, confidence intervals and p-value with reduced conditional bias.Fail fast! Statistical guidance to terminate a test early when it is unlikely to see a minimum desired effect.Optional stopping issues are solved with statistically rigorous rules for early stopping for efficacy.Great flexibility in monitoring results and decision making.classical fixed-sample tests ( 50-60% on average*) Significant efficiency improvements: 20%-80% less users needed vs. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |