Dichotomous thinking or binary thinking in statistics is the process of seeing a discontinuity in the possible values that a p-value can take during null hypothesis significance testing: it is either above the significance threshold (usually 0.05) or below. When applying dichotomous thinking, a first p-value of 0.0499 will be interpreted the same as a p-value of 0.0001 (the null hypothesis is rejected) while a second p-value of 0.0501 will be interpreted the same as a p-value of 0.7 (the null hypothesis is accepted). The fact that first and second p-values are mathematically very close is thus completely disregarded and values of p are not considered as continuous but are interpreted dichotomously with respect to the significance threshold. A common measure of dichotomous thinking is the cliff effect.[1] A reason to avoid dichotomous thinking is that p-values and other statistics naturally change from study to study due to random variation alone;[2][3] decisions about refutation or support of a scientific hypothesis based on a result from a single study are therefore not reliable.[4]
Dichotomous thinking is very often associated with p-value reading[5][6][7] but it can also happen with other statistical tools such as interval estimates.[1][8]
^Rosenthal, Robert; Gaito, John (1963). "The Interpretation of Levels of Significance by Psychological Researchers". The Journal of Psychology. 55 (1). Informa UK Limited: 33–38. doi:10.1080/00223980.1963.9916596. ISSN0022-3980.
^Nelson, Nanette; Rosenthal, Robert; Rosnow, Ralph L. (1986). "Interpretation of significance levels and effect sizes by psychological researchers". American Psychologist. 41 (11). American Psychological Association (APA): 1299–1301. doi:10.1037/0003-066x.41.11.1299. ISSN1935-990X.
^Besançon, Lonni; Dragicevic, Pierre (2019). "The Continued Prevalence of Dichotomous Inferences at CHI". Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. New York, New York, USA: ACM Press. pp. 1–11. doi:10.1145/3290607.3310432. ISBN978-1-4503-5971-9.
^Helske, Jouni; Helske, Satu; Cooper, Matthew; Ynnerman, Anders; Besancon, Lonni (2021). "Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect". IEEE Transactions on Visualization and Computer Graphics. 27 (8). Institute of Electrical and Electronics Engineers (IEEE): 3397–3409. arXiv:2002.07671. doi:10.1109/tvcg.2021.3073466. ISSN1077-2626. PMID33856998. S2CID233230810.