The N-Pact Factor, Replication, Power, and Quantitative Research in Adapted Physical Activity Quarterly

Click name to view affiliation

Jeffrey Martin
Search for other papers by Jeffrey Martin in
Current site
Google Scholar
PubMed
Close
and
Drew Martin
Search for other papers by Drew Martin in
Current site
Google Scholar
PubMed
Close
Restricted access

In the current study, a 20-year span of 80 issues of articles (N = 196) in Adapted Physical Activity Quarterly (APAQ) were examined. The authors sought to determine whether quantitative research published in APAQ, based on sample size, was underpowered, leading to the potential for false-positive results and findings that may not be reproducible. The median sample size, also known as the N-Pact Factor (NF), for all quantitative research published in APAQ was coded for correlational-type, quasi-experimental, and experimental research. The overall median sample size over the 20-year period examined was as follows: correlational type, NF = 112; quasi-experimental, NF = 40; and experimental, NF = 48. Four 5-year blocks were also analyzed to show historical trends. As the authors show, these results suggest that much of the quantitative research published in APAQ over the last 20 years was underpowered to detect small to moderate population effect sizes.

J. Martin is with the Div. of Kinesiology, Health, and Sport Studies, Wayne State University, Detroit, MI, USA. D. Martin is with Bloomfield Hills High School, Bloomfield Hills, MI, USA.

J. Martin (aa3975@wayne.edu) is corresponding author.
  • Collapse
  • Expand
  • Abelson, R.P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97(1), 129133. https://doi.org/10.1037/0033-2909.97.1.129

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ashford, S.J. (2013). Having scholarly impact: The art of hitting academic home runs. Academy of Management Learning & Education, 12(4), 623633. https://doi.org/10.5465/amle.2013.0090

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Atkinson, F., & Martin, J. (2020). Gritty, hardy, resilient, and socially supported: A replication study. Disability and Health Journal, 13(1), 100839. PubMed ID: 31519505 https://doi.org/10.1016/j.dhjo.2019.100839

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bacchetti, P., Wolf, L.E., Segal, M.R., & McCulloch, C.E. (2005). Ethics and sample size. American Journal of Epidemiology, 161(2), 105110. PubMed ID: 15632258 https://doi.org/10.1093/aje/kwi014

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Button, K.S., Ioannidis, J.P., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S., & Munafò, M.R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365376. PubMed ID: 23571845 https://doi.org/10.1038/nrn3475

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Camerer, C.F., Dreber, A., Holzmeister, F., Ho, T.H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637644. PubMed ID: 31346273 https://doi.org/10.1038/s41562-018-0399-z

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

  • Coles, N.A, Tiokhin, L., Scheel, A.M., Isager, P.M., & Lakens, D. (2018). The costs and benefits of replication studies. Behavioral and Brain Sciences, 41, e124. https://doi.org/10.1017/S0140525X18000596

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Earp, B.D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 621632. PubMed ID: 26042061 https://doi.org/10.3389/fpsyg.2015.00621

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Esposito, P.E., MacDonald, M., Hornyak, J.E., Ulrich, D.A. (2012). Physical activity patterns of youth with Down syndrome. Intellectual and Developmental Disabilities, 50(2), 109119. PubMed ID: 22642965 https://doi.org/10.1352/1934-9556-50.2.109

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Everett, J.A.C., & Earp, B.D. (2015). A tragedy of the (academic) commons: Interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Frontiers in Psychology, 6, 1152. PubMed ID: 26300832 https://doi.org/10.3389/fpsyg.2015.01152

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ezekiel, M. (1930). Methods of correlational analysis. Wiley.

  • Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891904. https://doi.org/10.1007/s11192-011-0494-7

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences, 115(11), 26282631. https://doi.org/10.1073/pnas.1708272114

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 11491160. PubMed ID: 19897823 https://doi.org/10.3758/BRM.41.4.1149

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fraley, R.C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One, 9(10), e109019. PubMed ID: 25296159 https://doi.org/10.1371/journal.pone.0109019

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fritz, C.O., Morris, P.E., & Richler, J.J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 218. https://doi.org/10.1037/a0024338

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haegele, J.A., Aigner, C.J., & Healy, S. (2019). Prevalence of meeting physical activity, screen-time, and sleep guidelines among children and adolescents with and without visual impairments in the United States. Adapted Physical Activity Quarterly, 36(3), 399405. PubMed ID: 31155913 https://doi.org/10.1123/apaq.2018-0130

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Halsey, L.G., Curran-Everett, D., Vowler, S.L., & Drummond, G.B. (2015). The fickle P value generates irreproducible results. Nature Methods, 12(3), 179185. PubMed ID: 25719825 https://doi.org/10.1038/nmeth.3288

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hansen, W.B., & Collins, L.M. (1994). Seven ways to increase power without increasing N. NIDA Research Monograph, 142, 184195. PubMed ID: 9243537

    • Search Google Scholar
    • Export Citation
  • Ioannidis J.P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. PubMed ID: 16060722 https://doi.org/10.1371/journal.pmed.0020124

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ivarsson, A., Andersen, M.B., Johnson, U., & Lindwall, M. (2013). To adjust or not adjust: Nonparametric effect sizes, confidence intervals, and real-world meaning. Psychology of Sport and Exercise, 14(1), 97102. https://doi.org/10.1016/j.psychsport.2012.07.007

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jager, L.R., & Leek, J.T. (2014). An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics, 15(1), 112. PubMed ID: 24068246 https://doi.org/10.1093/biostatistics/kxt007

    • Crossref
    • Search Google Scholar
    • Export Citation
  • John, L.K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524532. PubMed ID: 22508865 https://doi.org/10.1177/0956797611430953

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kerr, N.L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196217. PubMed ID: 15647155 https://doi.org/10.1207/s15327957pspr0203_4

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ketcheson, L.R., Centeio, E.E., Snapp, E.E., McKown, H.B., & Martin, J.J. (2021). Physical activity and motor skill outcomes of a 10-week intervention for children with intellectual and developmental disabilities ages 4–13: A pilot study. Disability and Health Journal, 14(1), 100952. PubMed ID: 32624452 https://doi.org/10.1016/j.dhjo.2020.100952

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Klein, R.A., Vianello, M., Hasselman, F., Adams, B.G., Adams, R.B., Alper, S., … Nosek, B.A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443490. https://doi.org/10.1177/2515245918810225

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lazic, S.E. (2018). Four simple ways to increase power without increasing the sample size. Laboratory Animals, 52(6), 621629. PubMed ID: 29629616 https://doi.org/10.1177/0023677218767478

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lilienfeld, S.O., & Strother, A.N. (2020). Psychological measurement and the replication crisis: Four sacred cows. Canadian Psychology, 61(4), 281288 https://doi.org/10.1037/cap0000236

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Little, T.D., Jorgensen, T.D., Lang, K.M., & Moore, E.W.G. (2014). On the joys of missing data. Journal of Pediatric Psychology, 39(2), 151162. PubMed ID: 23836191 https://doi.org/10.1093/jpepsy/jst048

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lloyd, M., Saunders, T.J., Bremer, E., & Tremblay, M.S. (2014). Long-term importance of fundamental motor skills: A 20-year follow-up study. Adapted Physical Activity Quarterly, 31(1), 6778. PubMed ID: 24385442 https://doi.org/10.1123/apaq.2013-0048

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Makel, M.C., Plucker, J.A., & Hegarty, B. (2012). Replications in psychology research how often do they really occur? Perspectives in Psychological Science, 7(6), 537542. https://doi.org/10.1177/1745691612460688

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Martin, J., Beasley, V., & Guerrero, M. (2019). Sport psychology research: Proper standards and limitations. In M.H. Anshel (Ed.), Handbook of sport and exercise psychology (pp. 17–40). American Psychological Association.

    • Search Google Scholar
    • Export Citation
  • Martin, J.J. (2017). Handbook of disability sport and exercise psychology. Oxford University Press.

  • Martin, J.J., Byrd, B., Watts, M.L., & Dent, M. (2015). Gritty, hardy, and resilient: Predictors of sport engagement and life satisfaction in wheelchair basketball players. Journal of Clinical Sport Psychology, 9(4), 345359. https://doi.org/10.1123/jcsp.2015-0015

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Martinez-Gutierrez, J.C., Leslie-Mazwi, T., Chandra, R.V., Ong, K.L., Nogueira, R.G., Goyal, M., … Hirsch, J.A. (2019). Number needed to treat: A primer for neurointerventionalists. Interventional Neuroradiology, 25(6), 613618. PubMed ID: 31248312 https://doi.org/10.1177/1591019919858733

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Maxwell, S.E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9(2), 147163. PubMed ID: 15137886 https://doi.org/10.1037/1082-989X.9.2.147

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Maxwell, S.E., Kelley, K., & Rausch, J.R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59(1), 537563. PubMed ID: 17937603 https://doi.org/10.1146/annurev.psych.59.103006.093735

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McClelland, G.H. (2000). Increasing statistical power without increasing sample size. American Psychologist, 55(8), 963964. https://doi.org/10.1037/0003-066X.55.8.963

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mill, J.S. (1950). Philosophy of scientific method. Hafner Publishing Co.

  • Moore, E.W.G., Lang, K.M., & Grandfield, E.M. (2020). Maximizing data quality and shortening survey time: Three-form planned missing data survey design. Psychology of Sport and Exercise, 51, 101701. https://doi.org/10.1016/j.psychsport.2020.101701

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Morin, K.H. (2016). Replication: Needed now more than ever. Journal of Nursing Education, 55(8), 423424. https://doi.org/10.3928/01484834-20160715-01

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mulkay, M., & Gilbert, G.N. (1986). Replication and mere replication. Philosophy of the Social Sciences, 16(1), 2137. https://doi.org/10.1177/004839318601600102

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nosek, B.A., & Lakens, D. (2014). A method to increase the credibility of published results. Social Psychology, 45(3), 137141. https://doi.org/10.1027/1864-9335/a000192

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pan, Z., & Kupper, L.L. (1999). Sample size determination for multiple comparison studies treating confidence interval width as random. Statistics in Medicine, 18(12), 14751488

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638641. https://doi.org/10.1037/0033-2909.86.3.638

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sanders, L.M., Hortobágyi, T., la Bastide-van Gemert, S., van der Zee, E.A., & van Heuvelen, M.J. (2019). Dose–response relationship between exercise and cognitive function in older adults with and without cognitive impairment: A systematic review and meta-analysis. PLoS One, 14(1), e0210036. PubMed ID: 30629631 https://doi.org/10.1371/journal.pone.0210036

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schweizer, G., & Furley, P. (2016). Reproducible research in sport and exercise psychology: The role of sample sizes. Psychology of Sport and Exercise, 23, 114122. https://doi.org/10.1016/j.psychsport.2015.11.005

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simmons, J., Nelson, L., & Simonsohn, U. (2011). Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 13591366. PubMed ID: 22006061 https://doi.org/10.1177/0956797611417632

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simons, D.J. (2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 7680. PubMed ID: 26173243 https://doi.org/10.1177/1745691613514755

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Thompson, B. (2006). Foundations of behavioral statistics: An insight-based approach. Guilford Press.

All Time Past Year Past 30 Days
Abstract Views 4241 897 8
Full Text Views 324 175 26
PDF Downloads 120 11 0