The use of patient-reported outcome measures (PROMs) has been shown to increase adherence to rehabilitation and improve communication between patients and their clinician through measurement of important subjective outcomes, such as quality of life and other psychological, sociological, and physiological factors.1,2 When identifying the most appropriate PROM for clinical use, the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) suggests that clinicians select PROMs with strong evidence of content and structural validity.3
Evidence of content validity is supported when the items from a PROM are relevant and logical for the outcome of interest. For example, a knee-specific PROM should only include items that measure knee-specific function. Evidence of structural validity can be supported through an examination of the relationship between examinees and their responses to items. Because of this, it is critical that item-level analyses be conducted on PROMs to determine whether items are measuring the content they were intended to measure. This relationship can be examined using classical test theory models or using more advanced models, such as item response theory (IRT) or Rasch models.4
The use of classical test theory models for item analysis is limited because the characteristics of the individual examinee and the characteristics of the items within a test cannot be separated from one another. This creates a generalizability concern where the quality of the measure is specific to the sample it was evaluated in. For example, significant differences have been identified across age groups and between sexes in normative scores for both the quality of life and sport/recreation subscales of the Knee Injury Osteoarthritis Outcome Score (KOOS).5 If the sample used to validate the KOOS was limited to a specific demographic group (age, sex, etc), the validity of the instrument would only be supported for that particular group. Another major limitation of classical test theory models is that error is considered consistent across all ranges of scores.
The IRT and Rasch model analyses, on the other hand, calculate item parameters and examinee characteristics independently, reducing the risk of bias associated with sample dependence. In addition, the standard error for IRT and Rasch models vary across the spectrum of scores, with separate errors reported for all possible ranges of scores. Another advantage of IRT and Rasch is that item parameters and examinee characteristics are placed on the same logit scale, allowing direct comparisons across different populations and groups. The logit scale is a standardized scale that is specific to IRT and Rasch models, similar to z scores or t scores seen commonly throughout the literature. A detailed discussion outlining the difference between IRT and Rasch model analysis is beyond the scope of this paper; however, Rasch model analysis is often considered more appropriate for instrument development.6
Rasch model analysis uses logistic modeling to examine the relationship between examinee characteristics and their responses to the items. This type of analysis allows for calculation of an item difficulty parameter and a person ability parameter. In the activities of daily living subscale of the KOOS, the item difficulty parameter for an item would reflect the difficulty associated with completing that task. The person ability parameter would be an estimation of the examinee’s total knee function. When the item difficulty for an item is the same as the person’s ability estimate, the examinee has a 50/50 chance of being able to perform the task. If the person ability estimate is higher than the item difficulty, they have a greater chance of completing the task. For polytomous items with multiple response categories like those commonly seen in PROM, such as the KOOS, the Rasch partial-credit model is used. When using the Rasch partial-credit model, a separate item difficulty, known as the item step difficulty or boundary location, is calculated between each response category. An item with 4 categories (eg, strongly agree, agree, disagree, strongly disagree) will have 3 item step difficulties. When person ability is equal to the step difficulty between 2 successive categories, the examinee is equally likely to respond within either category. For example, if the boundary location between 2 categories (strongly agree and agree) is 2.0 logits and an examinee with a person ability of 2.0 logits responds to the item, the examinee will have a 50% likelihood of responding in either the strongly agree or the agree categories. If the examinee’s ability is higher than 2.0 logits, then they will be more likely to respond in the strongly agree category compared with the agree category.
In a Rasch model analysis, separate fit statistics and item parameters are reported for each item. Fit statistics represent how well each item relates to the other items included in the measure (ie, model–data fit). Items with poor fit statistics are classified as “bad” items and are either revised or removed from the measure. Removal of these items serves 2 important benefits, as follows: first, removing “bad” items ensures that the remaining items are more closely related, giving a stronger representation of the outcome of interest; and second, removing items leads to shorter administration time, thus, lowering administrative burden on both the patient and the clinician.
A recent Rasch model analysis of several knee-specific PROM, which included the KOOS, the International Knee Documentation Committee Subjective Knee Form (IKDC), and the Marx Activity Rating Scale, reported that 54% of the items had poor model–data fit.7 This is problematic for populations where patients are likely to achieve high outcome scores (eg, student-athletes at the end of their rehabilitation), which reduces the amount of clinical value from these instruments. In the elite sports rehabilitation setting, following injury, athletes often experience a reduction in physical performance; however, despite the decrease in performance, the majority of injuries do not result in time loss from competition.8 For many clinicians, the goal of rehabilitation is to reduce pain levels and to improve patient function to their preinjury levels. To evaluate the effectiveness of rehabilitation and treatment outcomes, tools capable of assessing the upper limits of function are necessary. Because of the low clinical value of commonly used PROM, many clinicians feel that the administrative burden associated with using these measures is too high relative to the amount of information gained from their use.2 Performance-based outcomes are sometimes used as an alternative method for evaluating treatment efficacy; however systematic reviews of these instruments suggest that, in addition to poor measurement properties, these instruments are likely measuring 2 different constructs, making them a poor alternative for PROM.9
To address these limitations and improve assessment of knee-specific outcomes, a new measure was developed: the Patient-reported Outcomes Knee Assessment Tool (PROKAT). A primary goal during the development of the PROKAT was to reduce ceiling effects in higher functioning individuals. The PROKAT has the potential to improve assessment of knee-specific outcomes; however, the structural validity and internal consistency must be assessed to determine if the instrument has acceptable measurement properties for clinical use. Therefore, the purpose of this paper was to describe the development process of the PROKAT and to examine the measurement properties of the newly developed PROKAT in a sample of student-athletes using Rasch model analysis.
Study 1: Instrument Development and Pilot Testing
Pilot Test Instrument Development
Development and validation of the PROKAT occurred in 2 distinct phases. Study 1 outlines the development and pilot testing of the PROKAT. The goals during the pilot test were to evaluate the measure for readability and comprehension and to perform a preliminary assessment of the measurement properties. Data collection for the pilot test was approved by the institutional review board at Middle Tennessee State University prior to the start of data collection.
In a previous study examining the measurement properties of the KOOS, IKDC, and Marx Activity Rating Scale, it was identified that 30 of the items from these PROMs had acceptable measurement properties.7 Two pairs of items were duplicated across both the KOOS and IKDC, resulting in 28 unique items with acceptable measurement properties. Four of these initial 28 items were revised following review by the research team. One of the 4 items was revised to make the item more inclusive, while the other 3 were revised to make the items more specific and allow for the development of items involving more difficult functional tasks.
In the item “. . . difficulty with getting in/out of your car” the word car was replaced with vehicle to be more inclusive of alternative modes of transportation, such as trucks, motorcycles, sport utility vehicles, bicycles, or other alternative modes of transportation. The item “. . . difficulty with running” was changed to “. . . difficulty with jogging at 50% intensity.” The standardization of running pace was added because between-day variability of self-selected running speed can have low reliability.10 The intensity was selected to represent an activity of moderate intensity. Furthermore, as running speed increases, both peak pressure and peak force increase as well.11 The item “. . . difficulty with squatting” was changed to “. . . difficulty with performing double leg squats (body weight only)”. Finally, the item “. . . difficulty with jumping” was changed to “. . . difficulty with hopping repeatedly on your injured knee.” The changes for the previous 2 items were done to minimize variations in the interpretation of items by the participants and to allow for the development of future, more difficult items.
To reduce ceiling effects related to the assessment of knee function, 11 new items were developed and added to the existing item pool (see Table 1). Eight of the newly developed items were intended as more difficult versions of currently existing or recently modified items. Running at maximum speed, one of the newly assessed tasks should be more challenging than jogging at 50% intensity. Similarly, performing a single-leg squat should be more challenging than a double-leg or standard squat. In addition to potentially reducing ceiling effects, it was believed that these 2 items would allow an opportunity to further assess structural validity through an examination of item hierarchy (ie, individuals should report less difficulty with the easier task). In addition, single-leg tasks were added because many anterior cruciate ligament rehabilitation programs often include single-leg strengthening and balance-related exercises to minimize limb asymmetry.12 Studies indicate that many anterior cruciate ligament–deficient patients cannot perform stable single-leg squats.13 The remaining 3 items asked individuals to compare their current knee function in their injured knee to their noninjured knee, to their peers, or their knee function before the injury occurred (eg, worse, about the same, or better). The 3 comparison items were thought to improve and enhance the patient-centered focus of the assessment. The pilot version of the PROKAT, including all revisions, deletions, and newly developed items, contained 39 total items. An additional 3 open-ended items were added to obtain feedback from participants about the measure.
List of New Items Developed for Pilot Version of PROKAT (n = 11)
| Difficulty with picking a small object (eg, penny) off the floor? |
| Difficulty with lifting a large object off the floor? |
| Difficulty with performing single leg squats on your injured knee (body weight only)? |
| Difficulty with lower body resistance training (eg, weighted lunges or squats)? |
| Difficulty with running at maximum speed? |
| Difficulty with quickly changing direction while running (eg, agility drills)? |
| Difficulty with jumping forward and landing on your injured knee? |
| Difficulty with jumping to the side and landing on your injured knee? |
| Compared to your non-injured peers (eg, teammates or friends of similar athletic ability) how would you rate your overall knee function? |
| Compared to your healthy knee, how would you rate your injured knee’s overall level of function? |
| Compared to your non-injured peers (eg, teammates or friends of similar athletic ability) how would you rate your usual level of physical activity? |
Pilot Test Participant Recruitment and Procedures
Participants were recruited for the pilot test using convenience sampling methods (eg, flyer and word of mouth) from a single division III university. Participants were eligible for inclusion if they were current members of an NCAA division III sports team. During the initial phases of item development, it is recommended that, as soon as data from 30 to 50 participants are obtained, model–data fit and the rating scale structure of items be examined to make initial modifications/revisions to an instrument.14 Upon recruitment, the participants were instructed to complete an online survey, which contained basic demographic questions (age, sex, and sports team), the 39 pilot test PROKAT items, and 3 open-ended questions and which was hosted online using Google Forms (Google LLC, Mountain View, CA). Upon accessing, the survey participants read a brief explanation of the study and provided consent for completing the study by clicking “next.” Responses to all PROKAT items were required for submission of the survey.
Pilot Test Data Analysis
Descriptive statistics were calculated using SPSS (version 25.0; IBM Corp, Armonk, NY). A preliminary Rasch partial-credit model analysis was conducted using Xcalibre (version 4.0; Assessment Systems Corp, St. Paul, MN). Outcomes of interest for the pilot test included examination of model–data fit (infit and outfit statistics), item step difficulty, person ability parameters, and category function. The model–data fit for each item was evaluated using infit and outfit statistics. Infit and outfit statistics are reported as mean-square residuals, which are chi-square statistics divided by their degrees of freedom, so that they have a ratio-scale form, with an expected value of 1 and range from 0 to positive infinity. Mean-square residuals between 0.5 and 1.5 indicate that an item has acceptable fit.15 Values less than 0.5 indicate homogenization of scores, while values greater than 1.5 indicate large variability in scores.15
In the context of this study item, step difficulty represents the perceived difficulty of completing a task or the severity of a symptom. Person ability represents the examinee’s knee function. Both item step difficulty and person ability parameters range from negative to positive infinity and are expressed on a logit scale. To examine category function, a set of 8 guidelines was proposed by Linacre.16 The guidelines suggest that there should be a minimum of 10 responses for each possible response category within an item; the distribution of response options across categories should be relatively normal; the observed average person ability estimates (of the examinees who respond to a particular category) should advance monotonically with each category; the outfit mean square residuals should be less than 2 logits; the item step difficulties should increase (or decrease) consistently with each successive category; examinees respond in a manner that is expected, given their ability estimate; and finally, the item step difficulties between successive categories should increase by at least 1.4 logits but not more than 5.0 logits.16 Due to the small sample size used in the pilot test, not all criteria were considered during the pilot test.
Study 1: Pilot Test Results
A total of 32 healthy division III student-athletes (mean age = 20.78 [1.01], males = 56.30%) responded to all 39 items during the pilot test. The model–data fit and item step difficulty parameters for all 39 pilot test items are listed in Table 2. The examination of fit statistics for the 39 items indicated that model–data fit was within acceptable limits for 25 items (64%).
Pilot Test Rasch Calibration (n = 32)
| Item | Infit | Outfit | b0 | b1 | b2 |
|---|---|---|---|---|---|
| 1. How severe is your knee joint stiffness? | 0.955 | 0.684 | −1.162 | −3.627 | – |
| 2. How severe is your knee joint stiffness after sitting, lying, or resting later in the day? | 0.726 | 0.642 | 0.011 | – | – |
| 3. How often do you experience knee pain? | 0.594 | 0.600 | 1.178 | −0.543 | −2.975 |
| 4. Pain with twisting/pivoting on your knee? | 1.040 | 0.901 | −1.090 | – | – |
| 5. Pain with straightening knee fully? | 1.047 | 1.445 | −1.506 | – | |
| 6. Pain with bending knee fully? | 0.986 | 0.572 | −2.253 | – | – |
| 7. Pain with walking on a flat surface? | 0.962 | 0.418 | −3.434 | – | – |
| 8. Pain with going up or down stairs? | 1.096 | 1.034 | −1.854 | −3.323 | – |
| 9. Pain at night while in bed? | 0.912 | 0.432 | −2.942 | – | – |
| 10. Pain with sitting or lying? | 1.021 | 0.782 | −2.253 | – | – |
| 11. Difficulty with ascending/descending stairs? | 0.822 | 0.671 | −3.312 | −2.362 | – |
| 12. Difficulty with rising from sitting? | 0.843 | 0.425 | −2.819 | −2.733 | – |
| 13. Difficulty with getting in/out of your vehicle? | 0.787 | 0.252 | −3.434 | – | – |
| 14. Difficulty with standing? | 0.963 | 0.416 | −2.819 | −2.733 | – |
| 15. Difficulty with bending to the floor? | 0.701 | 0.335 | −3.175 | −2.128 | – |
| 16. Difficulty with picking up a small object (eg, penny) off the floor? | 1.092 | 0.665 | −4.213 | – | – |
| 17. Difficulty with lifting a large object off the floor? | 0.981 | 0.514 | −2.903 | – | – |
| 18. Difficulty with walking on a flat surface? | 0.971 | 0.438 | −3.434 | – | – |
| 19. Difficulty with sitting with your knee bent? | 0.895 | 0.558 | −1.606 | −3.443 | – |
| 20. Difficulty with performing double leg squats (body weight only?) | 1.066 | 0.758 | −2.566 | – | – |
| 21. Difficulty with performing single leg squats on your injured knee (body weight only)? | 0.543 | 0.400 | −1.452 | −1.950 | −2.921 |
| 22. Difficulty with lower body resistance training (eg, weight lunges, weighted squats)? | 0.880 | 0.626 | −1.246 | −2.770 | – |
| 23. Difficulty with jogging (approximately 50% intensity) | 1.206 | 0.711 | −2.253 | – | – |
| 24. Difficulty with running at maximum speed? | 1.286 | 0.862 | −2.002 | −2.402 | – |
| 25. Difficulty with quickly changing direction while running (eg, agility drills)? | 0.666 | 0.509 | −0.823 | −2.926 | – |
| 26. Difficulty with jumping forward and landing on your injured knee? | 1.020 | 0.617 | −1.745 | −1.549 | – |
| 27. Difficulty with jumping to the side and landing on your injured knee? | 0.732 | 0.522 | −1.293 | – | – |
| 28. Difficulty with hopping repeated on your injured knee? | 0.729 | 0.370 | −1.854 | −3.323 | – |
| 29. Difficulty with twisting/pivoting on your injured knee? | 0.589 | 0.436 | −1.030 | −2.854 | – |
| 30. Difficulty with kneeling on the front of your injured knee? | 1.389 | 1.327 | −2.063 | −1.361 | – |
| 31. How often are you aware of your knee problems? | 1.020 | 0.893 | −0.075 | −1.021 | −1.363 |
| 32. Have you modified your lifestyle to avoid potentially damaging activities? | 1.602 | 2.056 | −0.890 | −1.061 | – |
| 33. How much are you troubled with lack of confidence in your knee? | 0.876 | 0.453 | −1.874 | −1.917 | – |
| 34. What is the highest level of activity that you can perform without significant knee pain? | 0.869 | 0.404 | −1.960 | −2.695 | – |
| 35. What is the highest level of activity that you can perform without significant swelling in your knee? | 1.080 | 0.943 | −2.253 | – | – |
| 36. What is the highest level of activity that you can perform without significant giving way in your knee? | 0.907 | 0.325 | −3.434 | – | – |
| 37. Compared to your non-injured peers (eg, teammates or friends of similar athletic ability) how would you rate your overall knee function? | 0.861 | 0.838 | 0.751 | −1.745 | n/a |
| 38. Compared to your healthy knee, how would you rate your injured knee’s overall level of function? | 1.022 | 1.046 | 1.646 | −2.147 | n/a |
| 39. Compared to your non-injured peers (eg, teammates or friends of similar athletic ability) how would you rate your usual level of physical activity? | 1.153 | 1.211 | 0.906 | – | n/a |
Note: Item step difficulties and fit statistics are reported as logit values. Higher step difficulty values represent a more challenging task. Category options with 0 responses were merged using Xcalibre (version 4.0). No participants responded in the most extreme (fifth) response category; therefore, this column is not listed (as no data are present). Bolded rows represent misfitting items or items with category dysfunction. b0 represents the item step difficulty between the first response category and second response categories; b1 represents the item step difficulty between the second and third response categories; b2 represents the item step difficulty between the third and fourth response categories.
Study 1: PROKAT Revisions
Following a review of fit statistics, item step difficulties, examinee response patterns, and feedback from the pilot test results, several modifications were made to the original 39 pilot test items. Nine items from the pilot test were deleted, and 17 items were revised. A detailed summary of revisions for each item is provided in Table 3. One of the main concerns regarding participant responses, particularly among items assessing activities of daily living, was the lack of responses in the most severe categories. While the limited number of responses in severe categories is to be expected with a healthy sample population, it also suggests that the types of activities evaluated in these instruments may not be sufficiently challenging for athletes. This is further supported by the notion that limited responses in extreme categories were more prevalent in items involving very low difficulty tasks, such as walking on a flat surface. To address this concern, 2 additional items were developed. These items included: “pain while back pedaling (ie, running backward)” and “To what extent do you feel anxious about performing certain activities because of your injured knee.” In addition, it was identified that the change in the item step difficulties between the middle categories for some items was too small. This indicates that, for these items, the middle response options are not properly discriminating between examinees of different abilities. Reduction of the number of categories would provide greater separation between examinees and potentially increase the ability of the measure to distinguish between different levels of knee function. The reduction of category response options was also supported by data from a previous study, which suggested that using a 4-point Likert-type scale may give more meaningful and reliable data compared with the 5-point Likert-type scale7; therefore, the number of response options for items were reduced from 5 to 4 categories.
Pilot Test Item Revisions and Deletion List With Justification
| Item | Modification | Rational |
|---|---|---|
| 7. Pain with walking on a flat surface? | Item deleted | Item was too easy (30/32 in least severe category) and poor model–data fit. Although the item is related to ADL, the item is not sport related. As patient progresses through rehab, this item will likely become irrelevant early in the rehabilitation process. |
| 9. Pain at night while in bed? | Item deleted | Item was too easy (29/32 in least severe category) and poor model–data fit. Although the item is related to ADL, the item is not sport related. As patient progresses through rehab, this item will likely become irrelevant early in the rehabilitation process. |
| 11. Difficulty with ascending/descending stairs? | Item deleted | Item is almost identical to item number 8 (pain with going up or down stairs), and the language between the 2 was inconsistent. Participant feedback suggested that pain-related items would be more relevant to athletes because many athletes often perform activities despite pain due to high mental toughness. |
| 12. Difficulty with rising from sitting? | Item revised → Pain while rising from an armless chair? | Participants reported that the lack of clarity related to the type of chair could lead to some confusion. The degree of difficulty with rising from an armless chair is vastly different than one with arm rests, which can be used to assist with standing. In the original item, no participants responded to the most extreme category. Item changed from difficulty to pain based upon participant feedback. |
| 13. Difficulty getting in/out of your vehicle? | Item deleted | Item is too easy (30/32 in least severe category) and poor model–data fit. Although the item is related to ADL, the item is not sport related. As patient progresses through rehab, this item will likely become irrelevant early in the rehabilitation process. |
| 14. Difficulty with standing? | Item revised → Pain while standing for an extended period of time (minimum 1 hour)? | Participants reported that the lack of time frame associated with item could lead to ambiguity. Item revised to include a time frame. Time frame of at least 1 h was selected in an effort to increase difficulty of item. Item changed from difficult to pain based upon participant feedback. |
| 15. Difficulty with bending to the floor? | Item deleted | Item had poor model–data fit. In addition, it can be argued that bending to the floor is not directly related to the knee; bending is a hip-related movement which may or may not require the knee. |
| 16. Difficulty with picking up a small object (eg, penny) off the floor? | Item revised → Pain while picking up a small object (eg, penny) off the floor? | Item changed from difficulty to pain based upon participant feedback. |
| 17. Difficulty with lifting a large object off the floor? | Item revised → Pain while lifting a large object off the floor? | Item changed from difficulty to pain based upon participant feedback. |
| 18. Difficulty with walking on a flat surface? | Item deleted | Item was too easy (31/32 in least severe category) and poor model–data fit. Although the item is related to ADL, the item is not sport related. As patient progresses through rehab, this item will likely become irrelevant early in the rehabilitation process. |
| 19. Difficulty with sitting with your knee bent. | Item revised → Pain while sitting with your knee bent (at any degree) for an extended period of time (minimum 1 hour)? | Participants reported that the lack of time frame associated with item could lead to ambiguity. Item revised to include a time frame. Time frame of at least 1 h was selected because of the length of a typical college class. Item changed from difficulty to pain based upon participant feedback. |
| 20. Difficulty with performing double-leg squats (body weight only)? | Item revised → Pain while performing standard squats (body weight only)? | Participants felt that the use of the term “standard” was more appropriate than “double-leg” squats. Item changed from difficulty to pain based upon participant feedback. |
| 21. Difficulty with performing single leg squats (body weight only) on your injured knee? | Item revised → Pain while performing single-leg squats (body weight only) on your injured leg? | Item changed from difficulty to pain based upon participant feedback. Changed the word “knee” to “leg” to maintain consistency with other items. |
| 22. Difficulty with lower body resistance training (eg, weight lunges, weighted squats)? | Item revised → Pain while performing lower body resistance training (eg, lunges, step-ups, or squats)? | Item changed from difficulty to pain based upon participant feedback. Removed the terms “weight” and “weighted” from examples because of redundancy and to shorten item. |
| 23. Difficulty with jogging (approximately 50% intensity) | Item revised → Pain while jogging (approximately 50% intensity)? | Item changed from difficulty to pain based upon participant feedback. |
| 24. Difficulty with running at maximum speed? | Item revised → Pain while running at maximum speed? | Item changed from difficulty to pain based upon participant feedback. |
| 25. Difficulty with quickly changing direction while running (eg, agility drills)? | Item revised → Pain associated with changing direction while running (eg, agility drills)? | Item changed from difficulty to pain based upon participant feedback. |
| 26. Difficulty with jumping forward and landing on your injured knee? | Item revised → Pain with jumping forward and landing on your injured knee? | Item changed from difficulty to pain based upon participant feedback. |
| 27. Difficulty with jumping to the side and landing on your injured knee? | Item revised → Pain with jumping to the side and landing on your injured knee? | Item changed from difficulty to pain based upon participant feedback. |
| 28. Difficulty with hopping repeatedly on your injured knee? | Item revised → Pain with repeated high-impact jumps in place (such as jump knee tucks)? | Item had poor model–data fit. Increased item difficulty by changing intensity of jump in an effort to improve model–data fit and increase test difficulty. |
| 29. Difficulty with twisting/pivoting on your injured knee? | Item deleted | Item is almost identical to item number 4 (pain with twisting/pivoting on your injured knee). Participant feedback suggested that pain-related items would be more relevant to athletes because many athletes often perform activities despite pain due to high mental toughness. |
| 30. Difficulty with kneeling on your injured knee? | Item revised → Pain with kneeling on your injured knee? | Item changed from difficulty to pain based upon participant feedback. |
| 32. Have you modified your lifestyle to avoid potentially damaging activities? | Item revised → Have you modified your daily exercise or practice routines to avoid painful or potentially damaging activities? | Item had poor model–data fit (too much variability). Revised item to be more specific to athletic population. |
| 33. How much are you troubled with lack of confidence in your knee? | Item revised → To what extent are you troubled with lack of confidence in your knee when engaging in intense physical activity (such as during practice or games)? | Item had poor model–data fit. Revised item to be more specific to athletic population. |
| 34. What is the highest level of activity that you can perform without significant knee pain? | Item deleted | Item had poor model–data fit. |
| 36. What is the highest level of activity that you can perform without significant giving way in your knee? | Item deleted | Item had poor model–data fit. |
Interestingly, the item properties (eg, model–data fit, step difficulties) were often better for items assessing pain compared with items evaluating difficulty with performing a task. Sport psychology and sociology research reports that many athletes see “pain” as just a natural part of sports.17 This suggests that many athletes may perform a challenging task and report “no difficulty” in spite of pain. Thus, for athletes, being able to complete a task may be less relevant than the pain associated with completing a task. Although pain with activity and difficulty completing an activity are 2 different concepts, it is likely that items evaluating pain with completing an activity may be more appropriate for evaluating athletes’ knee function.
Following all revisions and the development of new items, the revised PROKAT included 32 items. The Flesh–Kincaid reading analysis suggests that the revised instrument requires a 10th-grade reading level, which is appropriate for college-aged athletes.18 Scoring for items with a 4-point Likert scale ranged from 0 to 3, while scoring for the 3 comparison items ranged from 0 to 2, with higher scores representing higher knee dysfunction.
Study 2: Rasch Calibration Methods
Rasch Calibration Participant Recruitment and Procedures
Participants were recruited from 3 separate universities and online, using social media (Twitter and Facebook). For the Rasch model analysis, a minimum sample size of 150 participants is required; however, sample sizes of at least 200 participants are preferred.19 As a result, data collection remained open until at least 200 complete responses were obtained. Upon recruitment, the participants were instructed to complete an online survey, which contained demographic questions (age, sex, height, weight, sport competition level, sport played, and current injury status) and the revised PROKAT. The demographic form and 32-item PROKAT were hosted online, using Google Forms. To increase recruitment and compliance, the participants who completed the study were offered the opportunity to win 1 of 4 $25 Visa gift cards to be dispersed at the end of the study. The participants who provided responses to all PROKAT items were eligible to be entered in a drawing that occurred at the conclusion of the study.
Rasch Calibration Data Analysis
Descriptive statistics were calculated using SPSS (version 25.0; IBM Corp). A Rasch partial-credit model analysis was conducted using Xcalibre (version 4.0; Assessment Systems Corp) to examine the psychometric properties of the revised PROKAT. The outcomes of interest were model–data fit, item step difficulties, person ability estimates, category function, test information function (TIF), item information function, the conditional standard error of measurement (CSEM), and Cronbach alpha. Acceptable ranges of model–data fit and category function were the same as those used during the pilot test. The TIF and item information function are graphical representations of how well the test performs across the spectrum of person ability estimates. The TIF is calculated from a summation of the item information function for all items in a measure. When the TIF is high for a specific ability range, it suggests that the test can measure those examinees well; conversely, when information functions are low, the test is a poor measure of ability for those examinees. The CSEM, which is calculated as the inverse of the TIF, is a representation of the precision of the measure. Acceptable CSEM for this study was set at 0.3. In addition, evidence of validity was examined through comparison of known-group differences in raw PROKAT scores between injured and noninjured athletes, using independent samples t test in SPSS (version 25.0; IBM Corp). The significance level was set at alpha equal to .05.
Study 2: Rasch Calibration Results
A total of 203 student-athletes (mean age = 21.46 [4.64], males = 54.70%) provided responses to all PROKAT items. Demographic characteristics for all participants have been provided in Table 4. The majority of participants (ie, 77.3%) self-reported as college-level division III athletes.
Participant Demographic Information—Phase 2 (n = 203)
| Variable | σ | n | % | |
|---|---|---|---|---|
| Age, y | 21.46 | 4.64 | ||
| Height, cm | 175.76 | 11.06 | ||
| Weight, kg | 79.85 | 18.99 | ||
| Sex | ||||
| Male | 111 | 54.68 | ||
| Female | 90 | 44.33 | ||
| Prefer not to say | 2 | 0.99 | ||
| Sport competition level | ||||
| College—division I | 18 | 8.90 | ||
| College—division II | 1 | 0.50 | ||
| College—division III | 157 | 77.30 | ||
| College—club sports | 2 | 1.00 | ||
| High school | 1 | 0.50 | ||
| Professional | 2 | 1.00 | ||
| Did not report | 22 | 10.80 | ||
| Sport | ||||
| Baseball | 27 | 13.30 | ||
| Basketball | 11 | 5.42 | ||
| Track and Field | 26 | 12.81 | ||
| Football | 32 | 15.76 | ||
| Wrestling | 9 | 4.43 | ||
| Rugby | 1 | 0.49 | ||
| Golf | 12 | 5.91 | ||
| Tennis | 10 | 4.93 | ||
| Soccer | 23 | 11.33 | ||
| Softball | 13 | 6.40 | ||
| Volleyball | 6 | 2.96 | ||
| Did not report | 33 | 16.26 | ||
| Current knee-related injury | ||||
| Yes | 36 | 17.70 | ||
| No | 167 | 82.30 | ||
Abbreviations: σ, standard deviation;
An examination of the infit and outfit statistics indicated that some of the items had poor model–data fit. The worst fitting item was removed, and the data were reanalyzed. This process was repeated until all items had acceptable infit and outfit statistics. Overall, the data fit the model well. Of the original 32 items examined on the PROKAT, 27 had acceptable infit and outfit statistics. The eliminated items have been listed in Table 5. Infit and outfit statistics and item step difficulties for all remaining items are listed in Table 6. The item step difficulties ranged from −4.74 to 1.89 logits. The person ability estimates ranged from −3.24 to 2.29 logits (mean = 0.00 [1.35]). Lower logit values for item step difficulties and person abilities estimates represented an easier task and lower knee function, respectively. In other words, an individual with high logit values for person ability will be more likely to endorse the least severe response options for an item (ie, no pain); conversely, an individual with lower logit values will be more likely to endorse the more severe response options. The 5 least difficult items were as follows: (1) pain with straightening of knee fully, (2) knee joint stiffness severity, (3) pain while lifting a large object off the floor, (4) pain while jogging at 50% intensity, and (5) pain with bending knee fully. The 5 most difficult items were as follows: (1) how often are you aware of your knee problems, (2) pain with kneeling on your injured knee, (3) compared with your noninjured peers, how would you rate your overall knee function, (4) compared with your healthy knee, how would you rate your knee’s overall level of function, and (5) pain while performing single-leg squats on your injured leg (body weight only).
List of Items Removed From PROKAT
| Item |
|---|
| • Pain while rising from an armless chair? |
| • Pain associated with changing direction while running (eg, agility drills)? |
| • Have you modified your daily exercise or practice routines to avoid painful or potentially damaging activities? |
| • Pain with repeated high-impact jumps in place (such as jump knee tucks)? |
| • Pain with jumping to the side and landing on your injured knee? |
Note: Each of the items listed in the table above were identified as having infit and/or outfit statistics outside of acceptable ranges.
Rasch Partial-Credit Model Results for PROKAT-Reduced Item Model
| Item | Infit | Outfit | b0 | b1 | b2 |
|---|---|---|---|---|---|
| 1. Pain while straightening knee fully? | 0.87 | 0.66 | −4.74 | −1.48 | −1.21 |
| 2. How severe is your knee joint stiffness? | 0.98 | 0.83 | −4.49 | −3.35 | −1.00 |
| 3. Pain while lifting a large object off of the floor? | 0.88 | 0.72 | −4.14 | −1.87 | −1.13 |
| 4. Pain while jogging at 50% intensity | 0.82 | 0.65 | −4.08 | −1.96 | −1.18 |
| 5. Pain with bending knee fully? | 1.04 | 0.80 | −3.99 | −2.20 | −0.93 |
| 6. Pain while going up or down stairs? | 0.71 | 0.56 | −3.96 | −1.62 | −0.87 |
| 7. Pain while sitting or lying? | 1.13 | 0.86 | −3.87 | −3.07 | −1.30 |
| 8. Pain while back pedaling (ie, running backward)? | 0.66 | 0.53 | −3.87 | −2.37 | −1.17 |
| 9. What is the highest level of activity that you can perform without significant swelling in your knee? | 0.92 | 1.13 | −3.80 | −2.33 | −3.09 |
| 10. How often do you experience knee pain? | 0.70 | 0.71 | −3.71 | −0.82 | 1.22 |
| 11. Pain while bending to the floor and picking up a small object (such as a penny) off the floor? | 0.97 | 0.73 | −3.65 | −2.16 | −1.22 |
| 12. Pain while performing standard squats (body weight only) | 0.75 | 0.67 | −3.41 | −1.80 | −0.70 |
| 13. Pain while twisting/pivoting on your knee? | 0.84 | 0.71 | −3.39 | −2.35 | −0.34 |
| 14. How severe is your knee joint stiffness after sitting, lying, or resting later in the day? | 1.05 | 0.93 | −3.31 | −3.29 | −0.95 |
| 15. Pain with jumping forward and landing on your injured knee? | 0.68 | 0.58 | −3.23 | −1.29 | −1.06 |
| 16. Compared to your non-injured peers (eg, teammates or friends of similar athletic ability) how would you rate your usual level of physical activity? | 1.41 | 1.39 | −3.18 | 0.61 | n/a |
| 17. Pain while running at maximum speed? | 0.85 | 0.76 | −3.07 | −1.82 | −0.49 |
| 18. Pain while standing for an extended period of time (minimum 1 hour)? | 1.13 | 1.00 | −3.01 | −1.39 | −0.13 |
| 19. Pain while performing lower-body resistance training exercises with weight (such as lunges, step-ups, or squats)? | 0.59 | 0.53 | −2.99 | −1.02 | −0.50 |
| 20. To what extent are you troubled with lack of confidence in your knee when engaging in intense physical activity (such as during practice or games)? | 0.75 | 0.62 | −2.99 | −2.49 | −2.00 |
| 21. To what extent do you feel anxious about performing certain activities because of your injured knee? | 0.81 | 0.56 | −2.57 | −3.20 | −1.73 |
| 22. Pain while sitting with your knee bent (at any degree) for an extended period of time (minimum 1 hour)? | 0.86 | 0.82 | −2.43 | −0.73 | −0.20 |
| 23. Pain while performing single-leg squats on your injured leg (body weight only) | 0.64 | 0.51 | −2.37 | −1.02 | −1.13 |
| 24. Compared to your healthy knee, how would you rate your injured knee’s overall level of function? | 1.01 | 1.00 | −2.24 | 1.89 | n/a |
| 25. Compared to your non-injured peers (eg, teammates or friends of similar athletic ability) how would you rate your overall knee function? | 1.03 | 1.00 | −2.23 | 0.79 | n/a |
| 26. Pain with kneeling on your injured knee? | 0.88 | 0.62 | −1.88 | −1.02 | −1.24 |
| 27. How often are you aware of your knee problems? | 0.92 | 0.98 | −1.84 | −1.22 | 0.47 |
Note: Item step difficulties and fit statistics are reported as logit values. Higher step difficulty values represent a more challenging task. b0 represents the item step difficulty between the first response category and second response categories; b1 represents the item step difficulty between the second and third response categories; b2 represents the item step difficulty between the third and fourth response categories; n/a represents no step difficulty is available for these items because there are only 3 possible response categories.
The participant’s scores on the PROKAT ranged from 0 to 64 (mean score = 17.11 [15.67]), with injured athletes (mean score = 39.25 [14.00]) scoring significantly higher (lower knee function) than noninjured athletes (11.93 [10.78]; t188 = 12.89; P < .01). Among the participants sampled in this study, only 3.9% obtained a score of 0 (ie, ceiling effect), while none of the participants obtained a 78, the maximum score (ie, floor effect). Cronbach alpha was .98. A frequency distribution of participant scores is displayed in Figure 1. The CSEM is displayed in Figure 2.
—This figure displays the distribution of participants’ total scores for the patient-reported outcomes knee assessment tool (PROKAT). Frequency represents the number of participants who achieved a given score. Scores range from 0 (no problems) to 78 (severe knee dysfunction).
Citation: Journal of Sport Rehabilitation 30, 2; 10.1123/jsr.2019-0264
—This figure displays the conditional standard error of measurement (CSEM) for the patient-reported outcomes knee assessment tool (PROKAT). The CSEM indicates the percentage of error associated with measuring person ability, expressed in decimal format. The dashed gray line is positioned at 0.3, which indicates the threshold for acceptable levels of measurement error. The solid black line indicates the degree of measurement error across the spectrum of person ability estimates measured in logits.
Citation: Journal of Sport Rehabilitation 30, 2; 10.1123/jsr.2019-0264
Discussion
Overall, the results from this study support the use of the PROKAT as a measure of knee function in student-athletes. Significant differences between the total scores of injured and noninjured student-athletes suggest that the measure may be suitable for distinguishing athletes of different degrees of knee function. The high Cronbach alpha (.98) coefficient also provides additional evidence of construct validity. Acceptable minimum thresholds for Cronbach alpha ranged from .70 to .95.20 One potential concern, however, with a very high Cronbach alpha coefficient (≥.90) is that there may be some redundancy across items.21 The presence of redundant items in the PROKAT is further supported by the low infit and outfit statistics reported for multiple items. This may be due to the fact that the majority of the participants in this sample were healthy student-athletes with no current injuries. This fact is also supported by the positively skewed (z = 2.45; P < .01) distribution of scores for the PROKAT. The inclusion of a higher proportion of injured athletes would likely result in more diverse response patterns, leading to a lower Cronbach alpha and higher logit values for infit and outfit statistics.
The final version of the PROKAT contained 27 items with acceptable model–data fit; only 4 of these items had poor category function. These 4 items included the following: (1) pain while performing single-leg squats on your injured leg (body weight only), (2) pain with kneeling on your injured knee, (3) to what extent do you feel anxious about performing certain activities because of your injured knee, and (4) what is the highest level of activity that you can perform without significant swelling in your knee? The disordered categories for each of the 4 items with category dysfunction all occurred between the moderate-severe and the mild-moderate categories.
For these 4 items, the number of response options may not be appropriate. Research investigating the number of optimal categories for rating scale questions provides conflicting evidence. While some studies suggest that increasing the number of category options can increase the reliability of an instrument,22 other studies have reported the opposite findings.23 The proportion of healthy athletes compared with injured athletes may have also been a contributing factor. The majority of the participants included in this study were generally healthy, with no current injuries (n = 167, 82.3%), resulting in few participants responding in the most severe category. Because of this, estimation of item step difficulties for the extreme categories may be biased. In the clinical setting, this measure is intended to be used on athletes who have recently been injured or are currently recovering from injury and, initially, will be expected to have greater functional limitations, making better use of the extreme categories.
Compared with alternative PROMs, the ceiling effects in the PROKAT were much smaller (3.9%) compared with ceiling effects found in other PROMs. The ceiling effects in healthy adults (n = 999) ranged from 33.9% to 75.0% across the subscales of the KOOS.5 Other studies, involving injured populations, have found ceiling effects of 28% in the KOOS24 and 15% for the IKDC.25 Ceiling effects are problematic for clinicians because, once the patient has obtained the maximum score, it is no longer possible to see improvements in function.
Although the ceiling effects are lower in the PROKAT compared with other PROMs, a potential concern is the decrease in measurement precision at higher levels of function. The measurement error for the PROKAT was within acceptable limits for participants, with person ability estimates below −0.25 logits. As function ability improved, measurement error increased. The measurement precision for the PROKAT was within acceptable limits for approximately 39% of the sample. The decrease in measurement precision for the higher levels of function suggests that the items included within the PROKAT are not sufficiently challenging or difficult enough to evaluate function ability as precisely in those with the highest levels of function. It is interesting to note, however, that 67% of the sample had a CSEM less than 0.40 logits. Another possible explanation for the lower precision is due to the low number of injured participants enrolled in the study. It is likely that inclusion of additional injured participants would result in more precise estimation of ability parameters.
An alternative solution for addressing ceiling effects is an evaluation of knee function using functional-based field testing, such as the 1-leg hop for distance test, timed single-leg hop test, Y-balance test, and percentage leg press. These functional-based tests, however, have poor correlations with PROMs, indicating that the 2 assessment strategies are measuring different constructs.26 Another limitation of functional-based field tests is the inability to assess psychosocial dimensions that are considered key components of patient function and overall well-being, as defined by the World Health Organization.27 While these functional-based field tests may provide useful supplemental information, they should not be considered as an alternative to traditional PROMs.
Two major advantages of this study were the use of advanced measurement theories (ie, Rasch measurement model) and the target sample (ie, athletes). Many of the current PROMs were not originally developed for athletic populations (eg, KOOS); those that were developed for athletes (eg, IKDC) were not calibrated using athletes; rather, the general population was used. As stated previously, studies have demonstrated key differences between athletes and nonathletes on measures of psychological and physiological well-being, key components of measuring patient function. Despite having strong content validity and the use of Rasch modeling, when the IKDC was tested in athletes, the items yielded poor model–data fit.7 These factors may be a contributing factor to the abundant ceiling effects identified in both the KOOS and IKDC.24,25
Despite the numerous benefits associated with the inclusion of PROMs in clinical practice, athletic trainers often report that administration time is a common barrier to implementation.2 Compared with the KOOS, which contain 42 items, the PROKAT only contains a total of 27 items, a 35% decrease in the total number of items, which should lead to decreased administration time. Furthermore, the PROKAT was developed and hosted online, using Google Forms. Google Forms is a free software program that allows for the development of survey instruments similar to Survey Monkey, which can be accessed from any electronic device. Because the PROKAT is hosted online, patients can complete the PROM before coming into the clinical setting, allowing time for the clinician to review the patient’s information and enhance communication between the patient and clinician. This has the potential to decrease clinician burden, thus, making the PROKAT an attractive option for clinicians.
Another commonly reported barrier is that many of the items in PROMs are not relevant to athletes. For example, on the KOOS, activities of daily living subscale items, such as difficulty with getting in/out of a car, going shopping, getting in/out of bath, heavy domestic duties, and light domestic duties, do not apply to athlete populations. Student-athletes often live on campus in dorms, where these questions may not apply. Many of the new items developed for the PROKAT included sportlike functional movements, which may help to enhance clinicians’ ability to evaluate patients. Single-leg squats, for instance, are often included in rehabilitation programs following anterior cruciate ligament injury28 to help improve knee stability and lower-extremity strength in the involved limb. The inclusion of this item, as well as the addition of other sportlike functional items, allows clinicians to assess the patient’s confidence related to performing these skills.
Limitations
This study was not without limitations. Modifications to the original measure were based upon information obtained during a small pilot study, where sample size was understandably small. As a result, inferences from the pilot study may be biased; however, many of the changes to the original measure were supported by data from a much larger sample in the full study. In addition, this study relied heavily on self-reported information collected anonymously from participants online. Due to the lack of physical contact with these participants, there was no way to verify the validity of the participants’ injury claims. The participants in this study were asked to report whether they were currently injured (yes/no) and to describe their injury (open-ended). While some of the participants reported valuable information related to the type and severity of their injury, most did not. The PROKAT, however, is intended to be a measure of general knee function, rather than an injury-specific instrument; therefore, the lack of this information did not influence the results of the study. Finally, although every effort was made to collect data on a diverse sample, the majority of the participants in this study were healthy student-athletes from division III sports. The high proportion of healthy participants compared with injured participants was a contributing factor for the skewed distribution of scores. This effect was minimal because of the use of non-sample-dependent analysis methods. In addition, comparisons of demographic characteristics across groups and proportions of injured and noninjured athletes within division I and division III yielded no significant differences (P < .05). Regardless, caution should be used when generalizing the results from this study to other groups of athletes. More studies should be conducted to investigate the quality of this measure in those groups.
Conclusions
The PROKAT is a newly developed PROM for assessing knee-specific outcomes in athletes. A major advantage of this study was the use of advanced measurement theory (ie, Rasch modeling) and the targeted population. Compared with commonly used PROM (eg, KOOS, IKDC-SKF), the PROKAT has greatly reduced ceiling effects in athletic populations. While future studies are needed to examine the reliability and responsiveness of the PROKAT throughout the rehabilitation process, the results from this study provide evidence of construct validity for the use of the PROKAT as a measure of knee-related function in athletic populations.
References
- 1.↑
Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Vol 323. Washington, DC: National Academic Press; 2001.
- 2.↑
Valier ARS, Jennings AL, Parsons JT, Vela LI. Benefits of and barriers to using patient-rated outcome measures in athletic training. J Athl Train. 2014;49(5):674–683. PubMed ID: 25098654 doi:10.4085/1062-6050-49.3.15
- 3.↑
Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–745. PubMed ID: 20494804 doi:10.1016/j.jclinepi.2010.02.006
- 4.↑
American Education Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. 7th ed. Washington, DC: American Educational Research Association; 2014.
- 5.↑
Williamson T, Sikka R, Tompkins M, Nelson BJ. Use of the knee injury and osteoarthritis outcome score in a healthy United States population. Am J Sports Med. 2016;44(2):440–446. PubMed ID: 26657571 doi:10.1177/0363546515616812
- 6.↑
Andrich D. Controversy and the Rasch model: a characteristic of incompatible paradigms? Med Care. 2004;42(suppl 1):I7–16. PubMed ID: 14707751.
- 7.↑
Farnsworth II J, Evans TA, Binkley H, Kang M. Evaluation of knee-specific patient-reported outcome measures using Rasch analysis [published online ahead of print February 7, 2020]. J Sport Rehabil. 1–8. doi:10.1123/jsr.2019-0263
- 8.↑
Dompier TP, Marshall SW, Kerr ZY, Hayden R. The National Athletic Treatment, Injury and Outcomes Network (NATION): methods of the surveillance program, 2011–2012 through 2013–2014. J Athl Train. 2015;50(8):862–869. PubMed ID: 26067620 doi:10.4085/1062-6050-50.5.04
- 9.↑
Terwee CB, Mokkink LB, Steultjens MPM, Dekker J. Performance-based methods for measuring the physical function of patients with osteoarthritis of the hip or knee: a systematic review of measurement properties. Rheumatology. 2006;45(7):890–902. PubMed ID: 16461441 doi:10.1093/rheumatology/kei267
- 10.↑
Queen RM, Gross MT, Liu H-Y. Repeatability of lower extremity kinetics and kinematics for standardized and self-selected running speeds. Gait Posture. 2006;23(3):282–287. PubMed ID: 15990310 doi:10.1016/j.gaitpost.2005.03.007
- 11.↑
Kemozek TW, Zimmer KA. Reliability and running speed effects of in-shoe loading measurements during slow treadmill running. Foot Ankle Int. 2000;21(9):749–752. doi:10.1177/107110070002100906
- 12.↑
Paterno MV, Ford KR, Myer GD, Heyl R, Hewett TE. Limb asymmetries in landing and jumping 2 years following anterior cruciate ligament reconstruction. Clin J Sport Med. 2007;17(4):258–262. PubMed ID: 17620778 doi:10.1097/JSM.0b013e31804c77ea
- 13.↑
Yamazaki J, Muneta T, Ju YJ, Sekiya I. Differences in kinematics of single leg squatting between anterior cruciate ligament-injured patients and healthy controls. Knee Surg Sports Traumatol Arthrosc. 2009;18(1):56. PubMed ID: 19693487 doi:10.1007/s00167-009-0892-z
- 14.↑
Fisher WP Jr. What scale-free measurement means to health outcomes research. Phys Med Rehabil State Art Rev. 1997;11(2):357–373.
- 15.↑
Wright BD, Linacre JM, Gustafson J, Martin-Lof P. Reasonable mean-square fit values. Rasch Meas Trans. 1994;8(3):370.
- 16.↑
Linacre JM. Optimizing rating scale category effectiveness. J Appl Meas. 2002;3(1):85–106. PubMed ID: 11997586.
- 17.↑
Young K, White P, McTeer W. Body talk: male athletes reflect on sport, injury, and pain. Sociol Sport J. 1994;11(2):175–194. doi:10.1123/ssj.11.2.175
- 18.↑
Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command, Millington TN: Institute for Simulation and Training, University of Central Florida. Research Branch; 1975.
- 19.↑
Sahin A, Anil D. The effects of test length and sample size on item parameters in item response theory. Educ Sci Theory Pract. 2017;17(1):321–335.
- 20.↑
DeVellis RF. Scale Development: Theory and Applications. Vol 26. Thousand Oaks, CA: Sage Publications; 2016.
- 21.↑
Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003;80(1):99–103. PubMed ID: 12584072 doi:10.1207/S15327752JPA8001_18
- 22.↑
Masters JR. The relationship between number of response categories and reliability of Likert-type questionnaires. J Educ Meas. 1974;11(1):49–53. doi:10.1111/j.1745-3984.1974.tb00970.x
- 23.↑
Matell MS, Jacoby J. Is there an optimal number of alternatives for Likert scale items? Study I: reliability and validity. Educ Psychol Meas. 1971;31(3):657–674. doi:10.1177/001316447103100307
- 24.↑
Roos EM, Toksvig-Larsen S. Knee injury and Osteoarthritis Outcome Score (KOOS)-validation and comparison to the WOMAC in total knee replacement. Health Qual Life Outcomes. 2003;1(1):17. doi:10.1186/1477-7525-1-17
- 25.↑
Fries J, Rose M, Krishnan E. The PROMIS of better outcome assessment: responsiveness, floor and ceiling effects, and Internet administration. J Rheumatol. 2011;38(8):1759–1764. PubMed ID: 21807798 doi:10.3899/jrheum.110402
- 26.↑
Burton C. Can the IKDC subjective patient outcome predict limb asymmetries in clinician reported outcomes? Free Communication Oral Presentations presented at National Athletic Trainers’ Association 68th Clinical Symposia & AT Expo; June 27, 2017. Houston, TX.
- 27.↑
World Health Organization. International Classification of Functioning, Disability and Health. Geneva, Switzerland: World Health Organization; 2001.
- 28.↑
Tagesson S, Öberg B, Good L, Kvist J. A comprehensive rehabilitation program with quadriceps strengthening in closed versus open kinetic chain exercise in patients with anterior cruciate ligament deficiency: a randomized clinical trial evaluating dynamic tibial translation and muscle function. Am J Sports Med. 2008;36(2):298–307. PubMed ID: 17940146 doi:10.1177/0363546507307867
