Sample ratio mismatch

In the design of experiments, a sample ratio mismatch (SRM) is a statistically significant difference between the expected and actual ratios of the sizes of treatment and control groups in an experiment. Sample ratio mismatches also known as unbalanced sampling[1] often occur in online controlled experiments due to failures in randomization and instrumentation.[2]

Sample ratio mismatches can be detected using a chi-squared test.[3] Using methods to detect SRM can help non-experts avoid making discussions using biased data.[4] If the sample size is large enough, even a small discrepancy between the observed and expected group sizes can invalidate the results of an experiment.[5][6]

Example

Suppose we run an A/B test in which we randomly assign 1000 users to equally sized treatment and control groups (a 50–50 split). The expected size of each group is 500. However, the actual sizes of the treatment and control groups are 600 and 400.

Using Pearson's chi-squared goodness of fit test, we find a sample ratio mismatch with a p-value of 2.54 × 10^-10. In other words, if the assignment of users were truly random, the probability that these treatment and control group sizes would occur by chance is 2.54 × 10^-10.[7]

References

1. Esteller-Cucala, Maria; Fernandez, Vicenc; Villuendas, Diego (2019-06-06). "Experimentation Pitfalls to Avoid in A/B Testing for Online Personalization". Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization. ACM. pp. 153–159. doi:10.1145/3314183.3323853. ISBN 978-1-4503-6711-0. S2CID 190007129.
2. Fabijan, Aleksander; Gupchup, Jayant; Gupta, Somit; Omhover, Jeff; Qin, Wen; Vermeer, Lukas; Dmitriev, Pavel (2019-07-25). "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments". Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM. pp. 2156–2164. doi:10.1145/3292500.3330722. ISBN 978-1-4503-6201-6. S2CID 196199621.
3. Nie, Keyu; Zhang, Zezhong; Xu, Bingquan; Yuan, Tao (2022-10-17). "Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection". Proceedings of the 31st ACM International Conference on Information & Knowledge Management. ACM. pp. 3391–3399. arXiv:2208.07766. doi:10.1145/3511808.3557087. ISBN 978-1-4503-9236-5. S2CID 251594683.
4. Vermeer, Lukas; Anderson, Kevin; Acebal, Mauricio (2022-06-13). "Automated Sample Ratio Mismatch (SRM) detection and analysis". The International Conference on Evaluation and Assessment in Software Engineering 2022. ACM. pp. 268–269. doi:10.1145/3530019.3534982. ISBN 978-1-4503-9613-4. S2CID 249579055.
5. Fabijan, Aleksander; Gupchup, Jayant; Gupta, Somit; Omhover, Jeff; Qin, Wen; Vermeer, Lukas; Dmitriev, Pavel (2019). "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners" (PDF). Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2156–2164. doi:10.1145/3292500.3330722. ISBN 9781450362016. S2CID 196199621.
6. Kohavi, Ron; Thomke, Stefan (2017-09-01). "The Surprising Power of Online Experiments". Harvard Business Review. ISSN 0017-8012. Retrieved 2023-05-19.
7. Vermeer, Lukas. "Frequently Asked Questions". SRM Checker. Retrieved 2022-09-15.