Patient-reported outcomes (PROs) are useful tools to assess patient function and monitor rehabilitation progress. The paper versions of PROs can be time-consuming due to manual grading, which may be a barrier to utilizing these outcomes.1 Digital methods of administration (MOA) have been shown to be just as accurate and reliable as their paper counterparts and eliminate the time required to score the instrument.2 The MOA has been shown to have no effect on the validity or reliability when comparing MOAs using instruments included in the Patient-Reported Outcomes Measurement Information System.2 Similar results have been reported in patients with rheumatoid arthritis and anterior cruciate ligament injury.3,4 Together, the research suggests that for musculoskeletal conditions, the time and cost benefits of using a digital MOA do not result in a trade-off to the psychometric properties of PROs.
The Foot and Ankle Ability Measure (FAAM) assesses function during activities of daily living (FAAM-ADL) and sports (FAAM-Sport).5 It is a common PRO that can be used during the initial evaluation of a patient as well as a metric to assess treatment success. The FAAM has been validated for use in assessing perceived disability in a variety of pathologies,6 making it a useful outcome in clinical trials. Researchers have established the minimally clinical important difference for the FAAM,5 and it has been used in clinical prediction rules for patients with chronic ankle instability (CAI).7,8 Herein, the gap between the clinic and the laboratory can be bridged with a time-efficient method of the FAAM, which can generate practice-based evidence to improve or refine their quality of care.
The purpose of this preliminary study was to examine the agreement between the paper format of the FAAM scales (FAAM-P) and a digital adaptation of the FAAM scales (FAAM-D). We further aimed to compare the test–retest reliability of the FAAM-P and FAAM-D. We hypothesize that the FAAM-D will agree with the FAAM-P and responses will be stable overtime on the digital MOA.
Methods
Study Design
We used a nonblinded, randomized, crossover observational design for this study.
Participants
We collected a convenience sample of 90 physically active individuals from the University of Nebraska at Omaha, recruited via digital advertisements, flyers, and e-mail communications. Participants were eligible if they were between the ages of 19 and 75 years, and English was their native language (Table 1). Informed consent was obtained from all participants prior to their completion, and this investigation was approved by the local institutional review board at the University of Nebraska at Omaha.
Participant Demographics for Each Analysis
| Agreement analysis N = 46 | Reliability analysisa N = 32 | |
|---|---|---|
| Sex | 19 male and 27 female | 14 male and 18 female |
| Age, y | 25.24 (7.36) | 25.63 (6.47) |
| Height, cm | 173.74 (10.62) | 175.22 (9.99) |
| Mass, kg | 75.30 (15.42) | 74.74 (15.63) |
| FAAM-P ADL, % | 90.92 (10.27) | 92.96 (7.86) |
| FAAM-D ADL, % | 90.59 (11.51) | 89.46 (12.77) |
| FAAM-P sport, % | 84.17 (11.85) | 84.43 (12.29) |
| FAAM-D sport, % | 83.94 (13.27) | 82.74 (16.25) |
Abbreviations: ADL, activities of daily living; FAAM-D, Foot and Ankle Ability Measure digital version MOA; FAAM-P, Foot and Ankle Ability Measure paper version. Note: Data are given as mean (SD).
aThe FAAM scores for the reliability analysis column represent the scores collected on the first day of testing.
Procedures
Participants reported to the first day (D1) and would be randomized via coin flip to complete the paper or digital MOA first (FAAM-P or FAAM-D). The FAAM-D was a digital adaptation of the FAAM-P on the Qualtrics platform (Qualtrics, Provo, UT), with all questions administered on an individual webpage per subscale of the FAAM (ie, 2 blocks of questions total). The FAAM-D was completed by participants on an iPad (Apple Inc, Cupertino, CA). This contained identical questions and response options to the original paper version,5 and it was graded using a script in Microsoft Excel (Redmond, WA). The grading of the FAAM-ADL and FAAM-Sports scales was identical to the originally developed version; a percentage was calculated for the maximum 84 and 32 points on the FAAM-ADL and FAAM-Sport, respectively.5 The maximum point pool was reduced and adjusted accordingly if an answer of “N/A” was recorded for any response field.5 Participants completed their first MOA and then an injury history questionnaire, which took 10 minutes to complete. Then, they completed the 2 FAAM scales on the other MOA. Participants returned 7 days after D1 and completed an identical randomized testing process (D2).
Statistical Analyses
Data were analyzed on the limb with the lowest score on the FAAM-P for each scale. Separate 2-way, random-effects intraclass correlation coefficient (ICC) was calculated to assess the absolute agreement between the 2 MOA for the FAAM-ADL and FAAM-Sport. We assessed test–retest reliability of the FAAM-P and FAAM-D using 2-way random-effects ICCs for consistency on the FAAM-ADL and FAAM-Sport scores recorded on D1 and D2. Our ICCs were interpreted as follows: <.5 (poor), .5–.75 (moderate), .75–.90 (good), and >.90 (excellent).9 ICCs were calculated using SPSS (version 24.0; IBM Inc, Armonk, NY). We also conducted bias estimates for the 2 MOA using the Bland–Altman Method in SigmaPlot (version 14.0; SYSTAT Software Inc, San Jose, CA).10 Alpha was set at .05 for all tests.
Results
Data from 44 participants were removed from the agreement analysis because they scored 100% on both MOA or data were missing. There was good agreement between the FAAM-P and FAAM-D. Results from our ICCs can be seen in Table 2, and our bias analysis can be seen in Figure 1. The Bland–Altman agreement analysis showed a mean bias of −0.14 (95% confidence interval, −6.97 to 6.69) for the FAAM-ADL and −0.23 (95% confidence interval, −13.13 to 12.68) for the FAAM-Sport.
ICC for Agreement and Test–Retest Reliability
| Absolute difference (SD) | ICC2,1 | ICC2,1 95% CI | |
|---|---|---|---|
| Agreement between the 2 MOA (n = 46) | |||
| FAAM-ADL | 3.23 (4.15) | .88* | .80 to.93 |
| FAAM-Sport | 4.56 (4.71) | .87* | .77 to.92 |
| Test–retest reliability of each measure (n = 32) | |||
| FAAM-P ADL | 3.06 (2.91) | .87* | .74 to.94 |
| FAAM-D ADL | 3.05 (5.40) | .89* | .78 to.95 |
| FAAM-P sport | 5.59 (7.64) | .71* | .49 to.85 |
| FAAM-D sport | 4.10 (5.19) | .91* | .83 to.96 |
Abbreviations: ADL, activities of daily living; CI, confidence interval; FAAM-D, Foot and Ankle Ability Measure digital version; FAAM-P, Foot and Ankle Ability Measure paper version; ICC, intraclass correlation coefficient; MOA, method of administration.
*Significant value (P < .05).
—Plots of the agreement and 95% CI between the 2 methods of administration. (A) Agreement between the FAAM-P and FAAM-D for the activities of daily living scale. (B) Bland–Altman analysis for bias and limits of agreement between MOA for the activities of daily living scale. (C) Agreement between the FAAM-P and FAAM-D for the sports scale. (D) Bland–Altman analysis for bias and limits of agreement between MOA for the sports scale. CI indicates confidence interval; FAAM-P, Foot and Ankle Ability Measure paper version; FAAM-D, Foot and Ankle Ability Measure digital version; MOA, method of administration.
Citation: Journal of Sport Rehabilitation 29, 7; 10.1123/jsr.2019-0321
Data from 58 participants were removed from the reliability analysis because they scored 100% on D1 and D2, and they indicated they had an injury between D1 and D2 or data were missing. There was good test–retest reliability of the FAAM-P for both scales. Comparatively, there was good-to-excellent test–retest reliability of the FAAM-D. These results can be seen in Table 2.
Discussion
Based on this preliminary analysis, it appears that a digital adaptation of the FAAM-ADL and FAAM-S provides similar response characteristics and stability overtime when compared with the validated paper format of the PRO.5 Specifically, we identified good agreement between the scales and good-to-excellent test–retest reliability in the FAAM-D.
This investigation has shown that the FAAM-D is comparable in responses and is stable overtime when compared with the FAAM-P. This is not an unexpected finding, as investigators have previously established high levels of agreement between paper and digital MOAs for PROs. Authors reported good agreement between the 2 MOA for the Tegner and Marx activity scales, Lysholm Knee Scale, and the International Knee Documentation Committee, with ICCs ranging from .65 to .79.3 The agreement between digital and paper MOA has also been established in patients with rheumatoid arthritis. It was found that a digitally delivered version of the Patient Global Assessment of Disease Activity, the Global Health score, and a visual analog scale had ICCs above .95 for all 3 measures.4 Thus, it may be possible to deliver the FAAM-D in a similar method, allowing for a long-term follow-up with a patient without them having to come into the clinic. Our ICCs suggest that the agreement between MOA may be stronger with the FAAM; however, we sampled physically active adults as opposed to patients. To better support the utility of the FAAM-D, a sample of an injured population may better reflect the stability of the FAAM-D and the variability that exists among patients.
While our ICC may suggest good agreement between MOA, our absolute differences and bias analysis indicate our results should be taken with caution. Table 2 shows the absolute differences between MOA. Each analysis has an average difference between MOA or overtime of around 3% to 5% on the FAAM, which indicates there is variation in how some individuals are scoring. Our bias estimates using the Bland–Altman method suggest a potential ceiling effect that is present in our sample and may explain the source of the absolute differences observed in our sample. Although our point estimates for bias suggest an acceptable level of agreement, the SD for both the FAAM-ADL and FAAM-Sport indicate a wider range of bias may be present. This is visible in participants with lower scores on either scale, where there tends to be larger differences between MOA as seen on the y-axis in Figure 1. This may reflect some systematic disagreement between the MOA, as it is easier to replicate answers of “No Difficulty” (ie, scores near 100%) on the FAAM than answers indicating difficulty during activities. Figure 1 suggests that there may be some discrepancy between the 2 MOA for the FAAM with lower agreement in lower scores. Using the FAAM in patient care will likely capture lower overall scores. Therefore, we suggest using a consistent MOA for all follow-up as mixed MOA follow-up and the FAAM-D have not yet been validated in injured populations.
A benefit of digital MOAs is that the clinician can have immediate access to their patient’s PRO score, significantly reducing the time burden. While PROs are common in physical therapy, athletic trainers, who also conduct musculoskeletal rehabilitation, may be less likely to utilize PROs due to time and cost barriers.1 The FAAM-D may be used during intake to screen a patient to help select the optimal treatment plan. The FAAM has been shown to be a critical determinant as to whether patients with CAI respond to balance training and manual therapies.7,8 Without screening, balance training, joint mobilizations, and massage are only successful at improving patient outcomes 40% to 50% of the time.7,8 Using a multimodal intake screening that incorporates the FAAM can increase the probability of a successful treatment between 70% and 99%.7,8 Our investigation showed that the FAAM-D agrees with the FAAM-P and is stable over time. The FAAM-D can be integrated into electronic medical records, meaning rehabilitation clinicians can capture these outcomes at the point-of-care. Interestingly, in our participants, the test–retest reliability of the FAAM-D was greater than that of FAAM-P, suggesting it may be better suited for serial testing or follow-up with patients. The FAAM is a common tool for assessing treatment efficacy in CAI research. A meta-analysis recently reported grade A evidence in support of conservative treatments (eg, balance training, resistance band exercises, joint mobilizations) improving health-related quality of life in patients with CAI, as measured by PROs such as the FAAM.11 Thus, digital MOAs may help bridge the gap between the laboratory and bedside if they were incorporated into practice-based evidence databases that can be used internally for quality improvement or to answer larger research questions.1
One limitation of this study is that we used a convenience sample of physically active adults opposed to those who are injured. The broad inclusion criteria allowed us to capture a larger sample size and compare the MOA. However, this may have generated the following 2 potential confounding factors: a ceiling effect mainly capturing higher scores on the FAAM and variation between MOA owing to differences in familiarity with technology. Furthermore, this means that our results should be taken with caution when used with patients given the larger variation in participants that scored lower on both subscales (Figure 1). However, this may benefit researchers as they often screen for study inclusion with the FAAM. Future studies should determine the validity and reliability of the FAAM-D in injured populations. We did not collect the time to complete each MOA of the FAAM; therefore, we are unable to examine if the amount of time it takes to complete the digital FAAM plays a role in the amount of error between the 2 MOA.
Conclusions
This preliminary investigation suggests that the responses on the 2 MOA agree with one another. We also identified that the stability overtime of the FAAM-D was greater than that of the paper version; thus, it may be better suited for follow-up testing to track treatment efficacy in single patients or effectiveness across a clinical practice. Future work should validate these claims in a larger sample size, as well as in pathology-specific populations.
Acknowledgments
This research was supported by a grant from the
References
- 1.↑
Bacon CEW, Eppelheimer BL, Kasamatsu TM, Lam KC, Nottingham SL. Athletic trainers’ perceptions of and barriers to patient care documentation: a report from the athletic training practice-based research network. J Athl Train. 2017;52(7):667–675. PubMed ID: 28574752 doi:10.4085/1062-6050-52.3.15
- 2.↑
Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware JE Jr. Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. J Clin Epidemiol. 2014;67(1):108–113. PubMed ID: 24262772 doi:10.1016/j.jclinepi.2013.07.016
- 3.↑
Nguyen J, Marx R, Hidaka C, Wilson S, Lyman S. Validation of electronic administration of knee surveys among ACL-injured patients. Knee Surg Sport Traumatol Arthrosc. 2017;25(10):3116–3122. doi:10.1007/s00167-016-4189-8
- 4.↑
Epis OM, Casu C, Belloli L, et al. Pixel or paper? Validation of a mobile technology for collecting patient-reported outcomes in rheumatoid arthritis. JMIR Res Protoc. 2016;5(4):e219. PubMed ID: 27852561 doi:10.2196/resprot.5631
- 5.↑
Martin RL, Irrgang JJ, Burdett RG, Conti SF, Van Swearingen JM. Evidence of validity for the foot and ankle ability measure (FAAM). Foot Ankle Int. 2005;26(11):968–983. PubMed ID: 16309613 doi:10.1177/107110070502601113
- 6.↑
Shazadeh Safavi P, Janney C, Jupiter D, Kunzler D, Bui R, Panchbhavi VK. A systematic review of the outcome evaluation tools for the foot and ankle. Foot Ankle Spec. 2019;12(5):461–470. PubMed ID: 30338697 doi:10.1177/1938640018803747
- 7.↑
Burcal CJ, Sandrey MA, Hubbard-Turner T, McKeon PO, Wikstrom EA. Predicting dynamic balance improvements following 4-weeks of balance training in chronic ankle instability patients. J Sci Med Sport. 2019;22(5):538–543. PubMed ID: 30501956 doi:10.1016/j.jsams.2018.11.001
- 8.↑
Wikstrom EA, McKeon PO. Predicting balance improvements following STARS treatments in chronic ankle instability participants. J Sci Med Sport. 2017;20(4):356–361. PubMed ID: 27840034 doi:10.1016/j.jsams.2016.09.003
- 9.↑
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. PubMed ID: 27330520 doi:10.1016/j.jcm.2016.02.012
- 10.↑
Giavarina D. Understanding bland Altman analysis. Biochem Med. 2015;25(2):141–151. doi:10.11613/BM.2015.015
- 11.↑
Powden CJ, Hoch JM, Hoch MC. Rehabilitation and improvement of health-related quality-of-life detriments in individuals with chronic ankle instability: a meta-analysis. J Athl Train. 2017;52(8):753–765. PubMed ID: 28704635 doi:10.4085/1062-6050-52.5.01
