Missing Data Reporting and Analysis in Motor Learning and Development: A Systematic Review of Past and Present Practices

in Journal of Motor Learning and Development

Click name to view affiliation

Priya PatelMichigan State University

Search for other papers by Priya Patel in
Current site
Google Scholar
PubMed
Close
*
,
Seungmin LeeMichigan State University

Search for other papers by Seungmin Lee in
Current site
Google Scholar
PubMed
Close
*
,
Nicholas D. MyersMichigan State University

Search for other papers by Nicholas D. Myers in
Current site
Google Scholar
PubMed
Close
*
, and
Mei-Hua LeeMichigan State University

Search for other papers by Mei-Hua Lee in
Current site
Google Scholar
PubMed
Close
*
Restricted access

Missing data incidents are common in experimental studies of motor learning and development. Inadequate handling of missing data may lead to serious problems, such as addition of bias, reduction in power, and so on. Thus, this study aimed to conduct a systematic review of the past (2007) and present (2017) practices used for reporting and analyzing missing data in motor learning and development. For this purpose, the authors reviewed 309 articles from five journals focusing on motor learning and development studies and published in 2007 and 2017. The authors carefully reviewed each article using a six-stage review process to assess the reporting and analyzing practices. Reporting of missing data along with reasons for their presence was consistently high across time, which slightly increased in 2017. Researchers predominantly used older methods (mainly deletion) for analysis, which only showed a small increase in the use of newer methods in 2017. While reporting practices were exemplary, missing data analysis calls for serious attention. Improvements in missing data handling may have the merit to address some of the major issues, such as underpowered studies, in motor learning and development.

The authors are with the Department of Kinesiology, Michigan State University, East Lansing, MI, USA.

Patel (patelp49@msu.edu) is corresponding author.
  • Collapse
  • Expand
  • Adolph, K.E., Cole, W.G., Komati, M., Garciaguirre, J.S., Badaly, D., Lingeman, J.M., . . . Sotsky, R.B. (2012). How do you learn to walk? Thousands of steps and dozens of falls per day. Psychological Science, 23(11), 13871394. doi:10.1177/0956797612446346

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Afifi, A.A., & Elashoff, R.M. (1966). Missing observations in multivariate statistics I. Review of the literature. Journal of the American Statistical Association, 61(315), 595604. doi:10.1080/01621459.1966.10480891

    • Search Google Scholar
    • Export Citation
  • Allison, P.D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112(4), 545. PubMed ID: 14674868 doi:10.1037/0021-843X.112.4.545

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452454. PubMed ID: 27225100 doi:10.1038/533452a

  • Bennett, D.A. (2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25(5), 464469. PubMed ID: 11688629 doi:10.1111/j.1467-842X.2001.tb00294.x

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Burton, A., & Altman, D.G. (2004). Missing covariate data within cancer prognostic studies: A review of current reporting and proposed guidelines. British Journal of Cancer, 91(1), 48. PubMed ID: 15188004 doi:10.1038/sj.bjc.6601907

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J., & Munafò, M.R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365376. PubMed ID: 23571845 doi:10.1038/nrn3475

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Camerer, C.F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., . . . Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637644. PubMed ID: 31346273 doi:10.1038/s41562-018-0399-z

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eekhout, I., de Boer, R.M., Twisk, J.W., de Vet, H.C., & Heymans, M.W. (2012). Missing data: A systematic review of how they are reported and handled. Epidemiology, 23(5), 729732. PubMed ID: 22584299 doi:10.1097/EDE.0b013e3182576cdb

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Enders, C.K. (2001a). The impact of nonnormality on full information maximum-likelihood estimation for structural equation models with missing data. Psychological Methods, 6(4), 352. doi:10.1037/1082-989X.6.4.352

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Enders, C.K. (2001b). The performance of the full information maximum likelihood estimator in multiple regression models with missing data. Educational and Psychological Measurement, 61(5), 713740. doi:10.1177/0013164401615001

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Enders, C.K. (2003). Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods, 8(3), 322. PubMed ID: 14596494 doi:10.1037/1082-989X.8.3.322

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Enders, C.K. (2010). Applied missing data analysis. New York, NY: Guilford press.

  • Enders, C.K. (2011). Missing not at random models for latent growth curve analyses. Psychological Methods, 16(1), 116. PubMed ID: 21381816 doi:10.1037/a0022640

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Enders, C.K. (2013). Analyzing structural equation models with missing data. In G.R. Hancock& R.O. Mueller, Structural equation modeling: A second course (2nd ed., pp. 493519). Charlotte, NC: Information Age Publishing.

    • Search Google Scholar
    • Export Citation
  • Enders, C.K., & Bandalos, D.L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 8(3), 430457. doi:10.1207/S15328007SEM0803_5

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Franchak, J.M., & Adolph, K.E. (2012). What infants know and what they do: Perceiving possibilities for walking through openings. Developmental Psychology, 48(5), 12541261. PubMed ID: 22390664 doi:10.1037/a0027530

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Garcia, T.P., & Marder, K. (2017). Statistical approaches to longitudinal data analysis in neurodegenerative diseases: Huntington’s disease as a model. Current Neurology and Neuroscience Reports, 17(2), 14. PubMed ID: 28229396

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Glasser, M. (1964). Linear regression analysis with missing observations among the independent variables. Journal of the American Statistical Association, 59(307), 834844.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gold, M.S., & Bentler, P.M. (2000). Treatments of missing data: A Monte Carlo comparison of RBHDI, iterative stochastic regression imputation, and expectation-maximization. Structural Equation Modeling, 7(3), 319355. doi:10.1207/S15328007SEM0703_1

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Graham, J.W., Cumsille, P.E., & Elek‐Fisk, E. (2003). Methods for handling missing data. In J. Schinka& W. Velicer (Eds.), Handbook of Psychology: Research Methods in Psychology (Vol. 2, pp. 87114). New York, NY: Wiley. doi:10.1002/0471264385.wei0204

    • Search Google Scholar
    • Export Citation
  • Graham, J.W., & Hofer, S.M. (2000). Multiple imputation in multivariate research. In T.D. Little, K.U. Schnabel, & J. Baumert (Eds.), Modeling longitudinal and multilevel data: Practical issues, applied approaches, and specific examples (pp. 201218, 269281). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

    • Search Google Scholar
    • Export Citation
  • Graham, J.W., & Schafer, J.L. (1999). On the performance of multiple imputation for multivariate data with small sample size. In R.H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 129). Thousand Oaks, CA: Sage.

    • Search Google Scholar
    • Export Citation
  • Harel, O., & Zhou, X.-H. (2007). Multiple imputation: Review of theory, implementation and software. Statistics in Medicine, 26(16), 30573077. PubMed ID: 17256804 doi:10.1002/sim.2787

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Horton, N.J., & Lipsitz, S.R. (2001). Multiple imputation in practice: Comparison of software packages for regression models with missing variables. The American Statistician, 55(3), 244254. doi:10.1198/000313001317098266

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ibrahim, J.G., & Molenberghs, G. (2009). Missing data methods in longitudinal studies: A review. Test, 18(1), 143. PubMed ID: 21218187 doi:10.1007/s11749-009-0138-x

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jamshidian, M., & Jalal, S. (2010). Tests of homoscedasticity, normality, and missing completely at random for incomplete multivariate data. Psychometrika, 75(4), 649674. PubMed ID: 21720450 doi:10.1007/s11336-010-9175-3

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jamshidian, M., Jalal, S., & Jansen, C. (2014). MissMech: An R package for testing homoscedasticity, multivariate normality, and missing completely at random (MCAR). Journal of Statistical Software, 56(1), 131. doi:10.18637/jss.v056.i06

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jeličić, H., Phelps, E., & Lerner, R.M. (2009). Use of missing data methods in longitudinal studies: The persistence of bad practices in developmental psychology. Developmental Psychology, 45(4), 1195. PubMed ID: 19586189 doi:10.1037/a0015665

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jones, M.P. (1996). Indicator and stratification methods for missing explanatory variables in multiple linear regression. Journal of the American Statistical Association, 91(433), 222230. doi:10.1080/01621459.1996.10476680

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Keselman, H.J., Huberty, C.J., Lix, L.M., Olejnik, S., Cribbie, R.A., Donahue, B., . . . Levin, J.R. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68(3), 350386. doi:10.3102/00346543068003350

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kim, K.H., & Bentler, P.M. (2002). Tests of homogeneity of means and covariance matrices for multivariate incomplete data. Psychometrika, 67(4), 609623. doi:10.1007/BF02295134

    • Crossref
    • Search Google Scholar
    • Export Citation
  • King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 95(1), 4969. doi:10.1017/S0003055401000235

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Little, R.J. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 11981202.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data. New York, NY: Wiley.

  • Lohse, K., Buchanan, T., & Miller, M. (2016). Underpowered and overworked: Problems with data analysis in motor learning studies. Journal of Motor Learning and Development, 4(1), 3758. doi:10.1123/jmld.2015-0010

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lohse, K., Shen, J., & Kozlowski, A.J. (2020). Modeling longitudinal outcomes: A contrast of two methods. Journal of Motor Learning and Development, 8(1), 121. doi:10.1123/jmld.2019-0007

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Munafò, M.R., Nosek, B.A., Bishop, D.V.M., Button, K.S., Chambers, C.D., Percie du Sert, N., . . . Ioannidis, J.P.A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 19. doi:10.1038/s41562-016-0021

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Muthén, B., Asparouhov, T., Hunter, A.M., & Leuchter, A.F. (2011). Growth modeling with nonignorable dropout: Alternative analyses of the STARD antidepressant trial. Psychological Methods, 16(1), 1733. PubMed ID: 21381817 doi:10.1037/a0022634

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Muthén, B., Kaplan, D., & Hollis, M. (1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52(3), 431462. doi:10.1007/BF02294365

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi:10.1126/science.aac4716

    • Search Google Scholar
    • Export Citation
  • Park, T., & Davis, C.S. (1993). A test of the missing data mechanism for repeated categorical data. Biometrics, 49(2), 631638. PubMed ID: 8369395 doi:10.2307/2532576

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Park, T., & Lee, S.-Y. (1997). A test of missing completely at random for longitudinal data with missing observations. Statistics in Medicine, 16(16), 18591871. PubMed ID: 9280038

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Peugh, J.L., & Enders, C.K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4), 525556. doi:10.3102/00346543074004525

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pigott, T.D. (2001). A review of methods for missing data. Educational Research and Evaluation, 7(4), 353383. doi:10.1076/edre.7.4.353.8937

  • Ranganathan, R., Tomlinson, A., Lokesh, R., Lin, T.-H., & Patel, P. (2020). A tale of too many tasks: Fragmentation of tasks in motor learning and a call for model task paradigms. Experimental Brain Research. Advance online publication. doi:10.1007/s00221-020-05908-6

    • Search Google Scholar
    • Export Citation
  • Rombach, I., Jenkinson, C., Gray, A.M., Murray, D.W., & Rivero-Arias, O. (2018). Comparison of statistical approaches for analyzing incomplete longitudinal patient-reported outcome data in randomized controlled trials. Patient Related Outcome Measures, 9, 197209. PubMed ID: 29950913 doi:10.2147/PROM.S147790

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roth, P.L. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47(3), 537560. doi:10.1111/j.1744-6570.1994.tb01736.x

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rubin, D.B. (1976). Inference and missing data. Biometrika, 63(3), 581592. doi:10.1093/biomet/63.3.581

  • Schafer, J.L. (1999). Multiple imputation: A primer. Statistical Methods in Medical Research, 8(1), 315. PubMed ID: 10347857

  • Schafer, J.L., & Graham, J.W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147177. PubMed ID: 12090408 doi:10.1037/1082-989X.7.2.147

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sullivan, T.R., White, I.R., Salter, A.B., Ryan, P., & Lee, K.J. (2018). Should multiple imputation be the method of choice for handling missing data in randomized trials? Statistical Methods in Medical Research, 27(9), 26102626. PubMed ID: 28034175

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tabachnick, B.G., & Fidell, L.S. (2012). Using multivariate statistics (6th ed.). Needham Heights, MA: Allyn & Bacon.

  • Thelen, E. (1981). Rhythmical behavior in infancy: An ethological perspective. Developmental Psychology, 17(3), 237257. doi:10.1037/0012-1649.17.3.237

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594604. doi:10.1037/0003-066X.54.8.594

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wood, A.M., White, I.R., & Thompson, S.G. (2004). Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clinical Trials, 1(4), 368376. PubMed ID: 16279275 doi:10.1191/1740774504cn032oa

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 1557 818 6
Full Text Views 158 7 0
PDF Downloads 56 11 0