On the Reproducibility of Power Analyses in Motor Behavior Research

Click name to view affiliation

Brad McKay Department of Kinesiology, McMaster University, Hamilton, ON, Canada

Search for other papers by Brad McKay in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-7408-2323 *
,
Mariane F.B. Bacelar Department of Kinesiology, Boise State University, Boise, ID, USA

Search for other papers by Mariane F.B. Bacelar in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-0388-1769
, and
Michael J. Carter Department of Kinesiology, McMaster University, Hamilton, ON, Canada

Search for other papers by Michael J. Carter in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-0675-4271 *
Restricted access

Recent metascience suggests that motor behavior research may be underpowered, on average. Researchers can perform a priori power analyses to ensure adequately powered studies. However, there are common pitfalls that can result in underestimating the required sample size for a given design and effect size of interest. Critical evaluation of power analyses requires successful analysis reproduction, which is conditional on the reporting of sufficient information. Here, we attempted to reproduce every power analysis reported in articles (k = 84/635) in three motor behavior journals between January 2019 and June 2021. We reproduced 7% of analyses using the reported information, which increased to 43% when we assumed plausible values for missing parameters. Among studies that reported sufficient information to evaluate, 63% reported using the same statistical test in the power analysis as in the study itself, and in 77%, the test addressed at least one of the identified hypotheses. Overall, power analyses were not commonly reported with sufficient information to ensure reproducibility. A nontrivial number of power analyses were also affected by common pitfalls. There is substantial opportunity to address the issue of underpowered research in motor behavior by increasing adoption of power analyses and ensuring reproducible reporting practices.

  • Collapse
  • Expand
  • Abt, G., Boreham, C., Davison, G., Jackson, R., Nevill, A., Wallace, E., & Williams, M. (2020). Power, precision, and sample size estimation in sport and exercise science research. Journal of Sports Sciences, 38(17), 19331935. https://doi.org/10.1080/02640414.2020.1776002

    • Search Google Scholar
    • Export Citation
  • Aust, F., & Barth, M. (2020). papaja: Prepare reproducible APA journal articles with R Markdown. https://github.com/crsh/papaja

  • Bacelar, M.F.B., Parma, J.O., Murrah, W.M., & Miller, M.W. (2022). Meta-analyzing enhanced expectancies on motor learning: Positive effects but methodological concerns. International Review of Sport and Exercise Psychology, 130. https://doi.org/10.1080/1750984X.2022.2042839

    • Search Google Scholar
    • Export Citation
  • Borg, D.N., Barnett, A., Caldwell, A.R., White, N., & Stewart, I. (2022). The bias for statistical significance in sport and exercise medicine. https://doi.org/10.31219/osf.io/t7yfc

    • Search Google Scholar
    • Export Citation
  • Carnegie, E., Marchant, D., Towers, S., & Ellison, P. (2020). Beyond visual fixations and gaze behaviour. Using pupillometry to examine the mechanisms in the planning and motor performance of a golf putt. Human Movement Science, 71, Article 102622. https://doi.org/10.1016/j.humov.2020.102622

    • Search Google Scholar
    • Export Citation
  • Carter, E.C., Kofler, L.M., Forster, D.E., & McCullough, M.E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144(4), 796815. https://doi.org/10.1037/xge0000083

    • Search Google Scholar
    • Export Citation
  • Chang, W. (2022). Extrafont: Tools for using fonts. https://CRAN.R-project.org/package=extrafont

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.

  • Correll, J., Mellinger, C., McClelland, G.H., & Judd, C.M. (2020). Avoid Cohen’s “small,” “medium,” and “large” for power analysis. Trends in Cognitive Sciences, 24(3), 200207. https://doi.org/10.1016/j.tics.2019.12.009

    • Search Google Scholar
    • Export Citation
  • Daou, M., Rhoads, J.A., Jacobs, T., Lohse, K.R., & Miller, M.W. (2019). Does limiting pre-movement time during practice eliminate the benefit of practicing while expecting to teach? Human Movement Science, 64, 153163. https://doi.org/10.1016/j.humov.2018.11.017

    • Search Google Scholar
    • Export Citation
  • DeBruine, L., Krystalli, A., & Heiss, A. (2021). faux: Simulation for factorial designs. https://CRAN.R-project.org/package=faux

  • Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 11491160. https://doi.org/10.3758/BRM.41.4.1149

    • Search Google Scholar
    • Export Citation
  • Fitzpatrick, P., de Jonge, E., & Warnes, G.R. (2019). Daff: Diff, patch and merge for data.frames. https://CRAN.R-project.org/package=daff

    • Search Google Scholar
    • Export Citation
  • Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641651.

    • Search Google Scholar
    • Export Citation
  • Harry, J.R., Lanier, R., Nunley, B., & Blinch, J. (2019). Focus of attention effects on lower extremity biomechanics during vertical jump landings. Human Movement Science, 68, Article 102521. https://doi.org/10.1016/j.humov.2019.102521

    • Search Google Scholar
    • Export Citation
  • Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), Article 33267. https://doi.org/10.1525/collabra.33267

  • Lakens, D., & Caldwell, A.R. (2021). Simulation-based power analysis for factorial analysis of variance designs. Advances in Methods and Practices in Psychological Science, 4(1), Article 2515245920951503. https://doi.org/10.1177/2515245920951503

    • Search Google Scholar
    • Export Citation
  • Lohse, K., Buchanan, T., & Miller, M. (2016). Underpowered and overworked: Problems with data analysis in motor learning studies. Journal of Motor Learning and Development, 4(1), 3758. https://doi.org/10.1123/jmld.2015-0010

    • Search Google Scholar
    • Export Citation
  • Maier, M., Bartoš, F., Stanley, T.D., Shanks, D.R., Harris, A.J.L., & Wagenmakers, E.-J. (2022). No evidence for nudging after adjusting for publication bias. Proceedings of the National Academy of Sciences, 119(31), Article e2200300119. https://doi.org/10.1073/pnas.2200300119

    • Search Google Scholar
    • Export Citation
  • McCrum, C., Beek, J., van Schumacher, C., Janssen, S., & Van Hooren, B. (2022). Sample size justifications in Gait & Posture. Gait & Posture, 92, 333337. https://doi.org/10.1016/j.gaitpost.2021.12.010

    • Search Google Scholar
    • Export Citation
  • McKay, B., Bacelar, M., Parma, J.O., Miller, M.W., & Carter, M.J. (2022). The combination of reporting bias and underpowered study designs have substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: A meta-analysis. PsyArXiv. https://doi.org/10.31234/osf.io/3nhtc

    • Search Google Scholar
    • Export Citation
  • McKay, B., Corson, A., Vinh, M.-A., Jeyarajan, G., Tandon, C., Brooks, H., Hubley, J., & Carter, M.J. (2022). Low prevalence of a priori power analyses in motor behavior research. Journal of Motor Learning & Development. https://doi.org/10.1123/jmld.2022-0042

    • Search Google Scholar
    • Export Citation
  • McKay, B., Hussien, J., Vinh, M.-A., Mir-Orefice, A., Brooks, H., & Ste-Marie, D.M. (2022). Meta-analysis of the reduced relative feedback frequency effect on motor learning and performance. Psychology of Sport and Exercise, 61, Article 102165. https://doi.org/10.1016/j.psychsport.2022.102165

    • Search Google Scholar
    • Export Citation
  • McKay, B., Yantha, Z.D., Hussien, J., Carter, M.J., & Ste-Marie, D.M. (2022). Meta-analytic findings in the self-controlled motor learning literature: Underpowered, biased, and lacking evidential value. Meta-Psychology, 6, 132. https://doi.org/10.15626/MP.2021.2803

    • Search Google Scholar
    • Export Citation
  • Mesquida, C., Murphy, J., Lakens, D., & Warne, J. (2022). Replication concerns in sports science: A narrative review of selected methodological issues in the field. SportRxiv. https://sportrxiv.org/index.php/server/preprint/view/127

    • Search Google Scholar
    • Export Citation
  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

    • Search Google Scholar
    • Export Citation
  • Qiu, W. (2021). powerMediation: Power/Sample size calculation for mediation analysis. https://CRAN.R-project.org/package=powerMediation

  • R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

    • Search Google Scholar
    • Export Citation
  • Rhoads, J.A., Daou, M., Lohse, K.R., & Miller, M.W. (2019). The effects of expecting to teach and actually teaching on motor learning. Journal of Motor Learning & Development, 7(1), 84105.

    • Search Google Scholar
    • Export Citation
  • Robinson, M.A., Vanrenterghem, J., & Pataky, T.C. (2021). Sample size estimation for biomechanical waveforms: Current practice, recommendations and a comparison to discrete power analysis. Journal of Biomechanics, 122, Article 110451. https://doi.org/10.1016/j.jbiomech.2021.110451

    • Search Google Scholar
    • Export Citation
  • Rudis, B., & Gandy, D. (2019). Waffle: Create waffle chart visualizations. https://gitlab.com/hrbrmstr/waffle

  • Twomey, R., Yingling, V., Warne, J., Schneider, C., McCrum, C., Atkins, W., Murphy, J., Medina, C.R., Harlley, S., & Caldwell, A. (2021). The nature of our literature: A registered report on the positive result rate and reporting practices in kinesiology. Communications in Kinesiology, 1(3), 117. https://doi.org/10.51224/cik.v1i3.43

    • Search Google Scholar
    • Export Citation
  • Uiga, L., Poolton, J.M., Capio, C.M., Wilson, M.R., Ryu, D., & Masters, R.S.W. (2020). The role of conscious processing of movements during balance by young and older adults. Human Movement Science, 70, Article 102566. https://doi.org/10.1016/j.humov.2019.102566

    • Search Google Scholar
    • Export Citation
  • Ushey, K. (2022). Renv: Project environments. https://CRAN.R-project.org/package=renv

  • Vohs, K.D., Schmeichel, B.J., Lohmann, S., Gronau, Q.F., Finley, A.J., Ainsworth, S.E., Alquist, J.L., Baker, M.D., Brizi, A., Bunyi, A., Butschek, G.J., Campbell, C., Capaldi, J., Cau, C., Chambers, H., Chatzisarantis, N.L.D., Christensen, W.J., Clay, S.L., Curtis, J., … Albarracín, D. (2021). A multisite preregistered paradigmatic test of the ego-depletion effect. Psychological Science, 32(10), 15661581. https://doi.org/10.1177/0956797621989733

    • Search Google Scholar
    • Export Citation
  • Westfall, J. (2015). PANGEA: Power analysis for general ANOVA designs. Unpublished Manuscript. https://osf.io/x5dc3

  • Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L.D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T.L., Miller, E., Bache, S.M., Müller, K., Ooms, J., Robinson, D., Seidel, D.P., Spinu, V., … Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686

    • Search Google Scholar
    • Export Citation
  • Wulf, G., & Lewthwaite, R. (2016). Optimizing performance through intrinsic motivation and attention for learning: The OPTIMAL theory of motor learning. Psychonomic Bulletin & Review, 23(5), 13821414. https://doi.org/10.3758/s13423-015-0999-9

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 920 757 33
Full Text Views 368 184 1
PDF Downloads 241 52 1