When are scientific findings deemed practically or theoretically relevant? A literature review on minimum-effect testing

Abstract

Null-hypothesis significance testing and p-values have been criticized for their focus on whether an effect is merely different from zero. To address this limitation, Murphy and Myors (1999) introduced minimum-effect testing to psychology researchers in the Journal of Applied Psychology approximately 25 years ago. Minimum-effect tests go beyond assessing whether an effect is statistically significantly different from 0; they also examine whether it is larger in magnitude than a predefined effect size (in a positive or negative direction) that is considered practically or theoretically meaningful. The usefulness of these tests depends largely on how researchers determine and justify the smallest effect size they consider meaningful, known as the smallest effect size of interest (SESOI). As minimum-effect tests are gaining popularity, we conducted a systematic literature review to examine how researchers in psychology apply minimum-effect tests, including how they define and justify their SESOI. We found that many studies relied on unstandardized effect sizes to define their SESOI. However, appropriate justifications for SESOIs were often missing, and in some cases, no justification was provided at all. Additionally, SESOIs were sometimes determined after data collection, or it was unclear when they were established. Furthermore, many studies did not account for minimum-effect testing in their power analyses. We offer several recommendations to improve the implementation of minimum-effect tests in psychological research.