The purpose of this Letter to the Editor is to discuss the reference accuracy of articles published in Volume 11 (2023) of the Journal of Motor Learning and Development (JMLD). Reference accuracy is defined as the completeness and correctness of bibliographic components, such as author names, article title, journal name, volume/issue, year of publication, pagination, and digital object identifier (DOI). An overall representation of reference accuracy is typically expressed as an error rate. An error rate is defined as the percentage of references in the reference list with at least one error and does not include examination of the accuracy of in-text citations.
The concept of reference accuracy in the context of scientific literature dates to at least 1881 in which the author writes “… the golden rule of verifying all his references …” (Billings, 1881, p. 220). Empirical research on reference accuracy began in the late 1950s (Kronick, 1958) and continues to present-day (Logan et al., 2023), including a tradition of scholars using letters to the editor to discuss reference accuracy (Crichton, 1989; McHugh, 1990; Narin et al., 2010; Nishina et al., 1995; Siebers & Holt, 2000).
There are numerous reasons why reference accuracy is important such, as crediting others for their work (George & Robbins, 1994; Hinchcliff et al., 1993; Oermann et al., 2001; Taylor, 1998), and providing the reader a means to find the original source (American Psychological Association, 2020; Key & Roland, 1977; Stull et al., 1991). A pattern of reference lists with errors may lead to a negative perception of a journal (George & Robbins, 1994; Lok et al., 2001; Sutherland et al., 2000). Reference accuracy is important for correct indexing of articles (Gupta & Babel, 2018). In turn, this contributes to accurate documentation of articles cited and influences the calculation of a journal’s impact factor (Davids et al., 2010; Fenton et al., 2000; Karabulut, 2017; Serenko et al., 2021). Therefore, a journal has a vested interest in ensuring reference accuracy in its published articles.
My interest in reference accuracy started as a graduate student at Auburn University. Dr. Mark Fischman was the instructor for a graduate course I completed on scientific writing. One of our readings included Stull et al. (1991). The authors found a 47% error rate across 973 citations in the 1988 and 1989 volumes of Research Quarterly for Exercise & Sport. The idea of reference accuracy lingered with me. Then, the COVID-19 pandemic hit in Spring 2020 and primary data collection with human participants was paused. I turned to reference accuracy to engage in scholarship.
I conducted a 30-year follow-up study to Stull et al. (1991). I verified 1,341 references from 50 articles across the years of 1999, 2009, and 2019 in Research Quarterly for Exercise & Sport and found an overall error rate of 40% (Logan, 2022). Although reference accuracy is well studied in health-related journals (Logan et al., 2023), only one other study was conducted exclusively within kinesiology-related journals (Zasa, 2015). The author found an overall error rate of 12% across the journals of American Journal of Sports Medicine, Exercise and Sport Sciences Reviews, Journal of Applied Physiology, Medicine & Science in Sports & Exercise, and Sports Medicine. However, the 12% error rate is based on verification of only 20 references per journal which is very limited in comparison to most studies on reference accuracy.
Then, I led a team that conducted a scoping review on reference accuracy studies published in peer-reviewed scholarly journals in any field of study (Logan et al., 2023). We found an overall error rate of 32.7% among the 105 included studies. Error rates were relatively similar across different fields of study. The most common errors were related to author names and article titles. Last, I had a bit of an off-the-wall idea to examine reference accuracy of studies that were included in the scoping review. The scoping review included empirical studies on reference accuracy studies where journals were selected and references within their published articles were verified. But how accurate are the reference lists of these scholars who conduct work on reference accuracy? That is, we wanted to understand if authors who conducted empirical studies about reference accuracy were better at reference accuracy compared to their peers. We verified 1,486 references of studies included in the scoping review and found a 21.2% overall error rate (Logan et al., 2024).
I thought that my scholarship with reference accuracy was finished until I became an Editorial Board Member of JMLD in August 2023 and transitioned to an Executive Editor in July 2024. I had a renewed interest in reference accuracy and thought I could use my knowledge on the topic to serve JMLD and its readers.
I replicated the methods of our previous work (Logan, 2022; Logan et al., 2024), with the addition that I also counted an error if a DOI was not included (and a DOI is assigned to the reference in question). I considered a missing DOI an error because its inclusion is a style requirement of JMLD that aligns with the Publication Manual of the American Psychological Association (7th ed.; American Psychological Association, 2020). An accurate DOI is also one of the most important aspects of a reference because other citation elements may be incorrect, yet an accurate DOI will allow readers to easily find the original source.
I verified references in JMLD across Issues 1, 2, and 3 published in 2023. I randomly selected eight references per article. Haylie L. Miller’s paper in Issue 3 included six references in total, so all were verified. This resulted in a total of 262 verified references which is sufficient to provide a representative snapshot of reference accuracy based on our previous work (see Figure 3 in Logan et al., 2023).
Findings indicated an overall error rate of 28.2% which is lower than Research Quarterly for Exercise and Sport (40%; Logan, 2022) and the scoping review about reference accuracy (32.7%; Logan et al., 2023), but higher than the descriptive study of reference accuracy of primary studies about reference accuracy (21.2%; Logan et al., 2024). The most common errors in JMLD were author names, DOIs, and article title. See Table 1 for results. This mostly aligns with our previous work that indicated the most common errors were author names and article titles (Logan, 2022; Logan et al., 2024). Sixteen out of 262 verified references included an error related to the DOI. The most common error related to DOIs was their omission.
Frequency, Percentage, Mean, SD, and 95% CIs Reported for Type of Reference Errors
| Type of error | Frequency (%) | Mean (SD) [95% CI] |
|---|---|---|
| At least one error related to author(s) names | 37 (14.1) | 0.1412 (0.35) [0.0988, 0.1837]a |
| Capitalization | 3 (1.1) | 0.0115 (0.11) [−0.0015, 0.0244] |
| Misspelled names | 4 (1.5) | 0.0153 (0.12) [0.0003, 0.0302]a |
| Incorrect or omitted initials | 23 (8.8) | 0.0878 (0.28) [0.0533, 0.1223]a |
| Omission of hyphens | 6 (2.3) | 0.0229 (0.15) [0.0047, 0.0411]a |
| Omission of names | 3 (1.1) | 0.0115 (0.12) [−0.0015, 0.0244] |
| Incorrect ordering of authors | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Incorrect names | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| Addition of authors not included in original work | 1 (0.4) | 0.0038 (0.62) [−0.0037, 0.0113] |
| Incorrect/omission of special characters | 2 (0.8) | 0.0076 (0.09) [−0.0030, 0.0182] |
| Any other error | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| At least one error related to article title | 12 (4.6) | 0.0458 (0.21) [0.0203, 0.0713]a |
| Omission of words | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Misspellings | 2 (0.8) | 0.0076 (0.09) [−0.0030, 0.0182] |
| Incorrect words | 5 (1.9) | 0.0191 (0.12) [0.0024, 0.0358]a |
| Addition of words | 0 (0.0) | 0.0 (0.00) [0.0, 0.0] |
| Omission of subtitles or not produced precisely | 0 (0.0) | 0.0 (0.00) [0.0, 0.0] |
| Incorrect punctuation | 5 (1.9) | 0.191 (0.14) [0.0024, 0.0358]a |
| At least one error related to journal title | 7 (2.7) | 0.0267 (0.16) [0.0071, 0.0464]a |
| Omission of words | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Misspellings | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| Incorrect words | 2 (0.8) | 0.0076 (0.09) [−0.0030, 0.0182] |
| Addition of words | 3 (1.1) | 0.0115 (0.12) [−0.0015, 0.0244] |
| Omission of subtitles | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| Incorrect punctuation | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| Any other discrepancy | 1 (0.4) | 0.0038 (0.6) [−0.0037, 0.0113] |
| At least one error related to volume | 2 (0.8) | 0.0076 (0.09) [−0.0030, 0.0182] |
| Incorrect | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Not included | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| At least one error related to issue | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Incorrect | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Not included | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| At least one error related to year | 6 (2.3) | 0.0029 (0.15) [0.0047, 0.0411]a |
| Incorrect | 6 (2.3) | 0.0029 (0.15) [0.0047, 0.0411]a |
| Not included | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| At least one error related to page numbers | 11 (4.2) | 0.0420 (0.20) [0.0175, 0.0664]a |
| Incomplete | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| Incorrect | 11 (4.2) | 0.0420 (0.20) [0.0175, 0.0664]a |
| Not included | 0 (0.0) | 0.0 (0.0) [0.0, 0.0] |
| At least one error related to DOI | 16 (6.1) | 0.0573 (0.23) [0.0289, 0.0856]a |
| Incorrect | 1 (0.4) | 0.0038 (0.06) [−0.0037, 0.0113] |
| Not included | 10 (3.8) | 0.0382 (0.19) [0.0148, 0.0615]a |
| Broken link | 5 (1.9) | 0.0191 (0.14) [0.0024, 0.0358]a |
| Total number of references with at least one error | 74 (28.2) | 0.2824 (0.45) [0.2276, 0.3373]a |
Note. Italic rows represent the number of references with at least one error at the main category level (i.e., does not account for if a reference had multiple errors in different main categories, such as author, article title, journal name, etc.). Roman rows represent the number of references with at least one error at the subcategory level (i.e., does not account for if a reference had multiple errors in the same subcategory such as capitalization of author names). Regardless of italic and roman rows, the denominator for all percentages is out of the number of verified references, which was 262. Bolded types of errors were classified as major (n = 44, 44.4%) and non-bolded types of errors were classified as minor (n = 55, 55.6% of errors) using 99 total errors as the denominator. DOI = digital object identifier; CI = confidence interval.
a95% CI does not include the value of 0.
Previous literature defines reference errors as either major or minor. depending on the perceived effect of the error on readers to find the original source (Logan, 2022; Logan et al., 2023). For example, errors related to punctuation are typically considered minor and errors related to the year of publication are considered major. Minor errors were more prevalent than major errors in JMLD which aligns with previous work (Logan, 2022; Logan et al., 2024).
The overall error rate for references in articles published in JMLD (28.2%) is better than reported in the scoping review (32.7%) of reference accuracy (Logan et al., 2023). I’ve learned throughout my scholarship on this topic that there will always be some number of errors in a reference list. In fact, I’ve found errors in my own reference lists after publication despite my best efforts. However, the goal of authors and JMLD is zero errors in reference lists. It is important that editors and authors are aware of reference accuracy as an important issue and understand that the most common types of errors are related to author names (especially their initials) and article titles. This awareness of common errors may lead to facilitation of an improved overall error rate for JMLD if these citation components are carefully reviewed for accuracy at the page proof stage of publication.
References
American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.).
Billings, J.S. (1881). An address on our medical literature. British Medical Journal, 2(1076), 262–268.
Crichton, R.R. (1989). Complacent reviewing. Nature, 337(6203), Article 110.
Davids, J.R., Weigl, D.M., Edmonds, J.P., & Blackhurst, D.W. (2010). Reference accuracy in peer-reviewed pediatric orthopaedic literature. Journal of Bone & Joint Surgery, 92(5), 1155–1161.
Fenton, J.E., Brazier, H., De Souza, A., Hughes, J.P., & McShane, D.P. (2000). The accuracy of citation and quotation in otolaryngology/head and neck surgery journals. Clinical Otolaryngology & Allied Sciences, 25(1), 40–44.
George, P.M., & Robbins, K. (1994). Reference accuracy in the dermatological literature. Journal of the American Academy of Dermatology, 31(1), 61–64.
Gupta, V.K., & Babel, P. (2018). Accuracy of references in journal literature of medical sciences: A review. IRA-International Journal of Management & Social Sciences, 12(3), 62–72.
Hinchcliff, K.W., Bruce, N.J., Powers, J.D., & Kipp, M.L. (1993). Accuracy of references and quotations in veterinary journals. Journal of the American Veterinary Medical Association, 202(3), 397–400.
Karabulut, N. (2017). Inaccurate citations in biomedical journalism: Effect on the impact factor of the American Journal of Roentgenology. American Journal of Roentgenology, 208(3), 472–474.
Key, J.D., & Roland, C.G. (1977). Reference accuracy in articles accepted for publication in the Archives of Physical Medicine and Rehabilitation. Archives of Physical Medicine and Rehabilitation, 58(3), 136–137.
Kronick, D.A. (1958). Literature citations, a clinico-pathological study, with the presentation of three cases. Bulletin of the Medical Library Association, 46(2), 219–223.
Logan, S.W. (2022). Reference accuracy in Research Quarterly for Exercise and Sport: A 30-year follow-up to Stull et al. (1991). Research Quarterly for Exercise and Sport, 93(2), 401–411.
Logan, S.W., Hussong-Christian, U., Case, L., & Noregaard, S. (2023). Reference accuracy of primary studies published in peer-reviewed scholarly journals: A scoping review. Journal of Librarianship and Information Science. Advance online publication.
Logan, S.W., Hussong-Christian, U., Case, L., & Noregaard, S. (2024). Reference accuracy of primary studies about reference accuracy: A descriptive study. Journal of Librarianship and Information Science. Advance online publication.
Lok, C.K.W., Chan, M.T.V., & Martinson, I.M. (2001). Risk factors for citation errors in peer-reviewed nursing journals. Journal of Advanced Nursing, 34(2), 223–229.
McHugh, J.L. (1990). Literature citations: A comedy of errors. BioScience, 40(5), Article 338.
Narin, S., Koçak, F., Ozalevli, S., & Ilgin, D. (2010). Reference accuracy in four respiratory medical journals. Respiration, 80(2), 171–172.
Nishina, K., Asano, M., Mikawa, K., Maekawa, N., & Obara, H. (1995). Accuracy of references in Paediatric Anaesthesia. Pediatric Anesthesia, 5(2), 142–144.
Oermann, M.H., Cummings, S.L., & Wilmes, N.A. (2001). Accuracy of references in four pediatric nursing journals. Journal of Pediatric Nursing, 16(4), 263–268.
Serenko, A., Dumay, J., Hsiao, P.-C.K., & Choo, C.W. (2021). Do they practice what they preach? The presence of problematic citations in business ethics research. Journal of Documentation, 77(6), 1304–1320.
Siebers, R., & Holt, S. (2000). Accuracy of references in five leading medical journals. The Lancet, 356(9239), Article 1445.
Stull, G.A., Christina, R.W., & Quinn, S.A. (1991). Accuracy of references in Research Quarterly for Exercise and Sport. Research Quarterly for Exercise and Sport, 62(3), 245–248.
Sutherland, A.G., Craig, N., Maffulli, N., Brooksbank, A., & Moir, J.S. (2000). Accuracy of references in the orthopaedic literature. Journal of Bone and Joint Surgery. British Volume, 82-B(1), 9–10.
Taylor, M.K. (1998). The practical effects of errors in reference lists in nursing research journals. Nursing Research, 47(5), 300–303.
Zasa, M. (2015). The accuracy of references in five sport science journals. Science & Sports, 30(1), e31–e33.
