Key Points
- ▸Male athletes reported significantly more symptoms in a face-to-face interview using the SCAT3 items compared to the computerized self-report using the ImPACT symptom scale.
- ▸Females reported more symptoms and a greater severity of symptoms than did males.
- ▸Examiner sex did not affect performance in the face-to-face interview for male or female athletes.
When baseline protocols are used for assessment of sport-related concussion, the posttrauma evaluation compares posttrauma with precompetition performance in multiple areas that usually include computerized neurocognitive measures, balance ability, and concussion-related symptoms.1 Accurate symptom reporting at baseline is important for proper interpretation of the athlete’s posttrauma symptom report.2 Moreover, in making return-to-play recommendations, athletic trainers were found to rely, in the vast majority of cases, upon symptom reports.3 In considering the various ways in which prevailing symptoms are ascertained, the following appear to be inclusive of standard practices: (a) athlete independently reporting via completion of a paper and pencil symptom scale such as the postconcussion symptom scale (PCSS)4; (b) athlete independently self-reporting via completion of computerized symptom checklist as with the Immediate Post-Concussion Assessment and Cognitive Test (ImPACT)5; (c) completion of PCSS items in response to inquiry by clinician in a structured interview format, and (d) responses to clinician questioning in unstructured interview format. Despite global cautions about significant biasing that may occur across modes of ascertaining subject information,6 there have been very few studies or comments in the sport concussion literature.
History of concussion, motivation to perform, and sex of examinee have been shown to impact baseline self-report of symptoms.2,7–9 In addition, examination of baseline symptom reporting in collegiate football and hockey players found that number and severity of symptoms differed significantly based on how the symptom information was collected.10 In that cross-sectional study, one group of the 117 male athletes completed the PCSS on their own in a group setting, while the other group responded to the same items individually in an interview format. The athletes reported more symptoms on the independently completed PCSS versus the structured interview. Researchers noted also that athletes (all male) reported more symptoms when the examiner was female versus male, although the group sizes for this comparison were fairly small (N = 20–30).10
With technological advancements, computer-based assessments have been integrated into the process of concussion evaluation and management.11 Since ImPACT is by far the most widely used neurocognitive test in concussion management, the vast bulk of studies on reliability, validity, and predictive power have been conducted using ImPACT. The broadest conclusions summarized in recent reviews and meta-analyses were that assessments such as ImPACT converge reasonably well with standard tests when administered in controlled conditions by a trained examiner and ImPACT has poor to moderate reliability (which obviously limits validity) and should not be the sole measure used in concussion management.12 Despite this dour summary, ImPACT has been used widely (with apparent success), and methods such as the aggregate baseline have been recommended to improve reliability.13
In the present study, not unlike Krol and colleagues,10 we aimed to assess the convergent validity of two different methods of ascertaining symptom profiles. Unlike the Krol study, we compared data from a structured interview inquiry about symptoms with an examiner-free symptom self-report in the same athletes. Thus, we had a real-time, short latency observation of the two methods for comparison utilizing the same examinee to assess for change. Discovering differences in symptom report based upon standard methods of assessment has significant implications not only on the ability to detect concussion through pre- and postsymptom comparison, but also in the ability of psychologists, physicians, and athletic training staff to identify individuals who may be at risk for worse cognitive outcomes after a concussion. High baseline symptom reporting may predict which athletes are at greater risk acutely following a concussion.14
We also analyzed whether the sex of the athlete interacted with assessment type and sex of the examiner. Despite the lessons from social psychology research that sex may represent a ubiquitous demand characteristic in social and structured settings, there has been little systematic study and report of such interactions in the concussion literature other than the frequently reported phenomenon that females generally report more symptoms at baseline7 and postconcussion.15,16 Therefore, the present research aimed to investigate not only the differences in symptom reporting between examiner-based and computerized assessments within a large sample of male and female collegiate athletes, but also the interaction of athlete sex with examiner sex.
Method
Participants
Participants were 754 collegiate student-athletes (463 males [61.4%] and 291 females [38.6%]) who were administered the assessment battery by two male examiners and 17 female examiners before the start of the 2015–2016 or 2016–2017 athletic seasons. The age of participants ranged from 17 to 27 (M = 19.53; SD = 1.59) with a mean education level of 13.32 years (SD = 1.27; range = 12–17 years). The mean number of previous concussions was 0.36 (SD = .80). Collegiate athletic teams were comprised of 14 contact and noncontact sports. Figure 1 displays the number of participants from each team. All participants indicated by signature that they consented to participate.
—The number of participants from each collegiate athletic team.
Citation: International Journal of Athletic Therapy and Training 24, 2; 10.1123/ijatt.2018-0028
Instrumentation
The Sport Concussion Assessment Tool–3rd Edition (SCAT3)17 is a standardized instrument for evaluating athletes for concussion. The symptom evaluation portion of the SCAT3 consists of 22 symptoms that are commonly experienced by athletes after concussion. The symptom checklist is based on a 7-point scale, with 0 and 6 reflecting the anchor points. Zero represents the absence of a symptom, 1 represents mild severity, and 6 represents severe severity. Athletes in the current study completed the scale prior to the start of their competitive seasons to indicate the frequency at which they experience typical postconcussion symptoms on a day-to-day basis. The severity of the symptoms was added (0–6) for each item to obtain a total severity score.
The ImPACT is a computerized battery of tests designed specifically to detect changes in neurocognitive functioning following sport-related concussion. In addition to collecting basic demographic measures, descriptive data, and neurocognitive functioning, ImPACT includes the PCSS.4 Similar to the SCAT3 symptom scale, this 22-item symptom checklist employs a 7-point scale, with 0 and 6 reflecting the anchor points. Both the SCAT3 and the ImPACT symptom scales have a common origin in the Pittsburgh Steelers Post-Concussion Scale.18
Tasks and Procedures
As part of their precompetition medical testing, athletes experienced a 30-min concussion education session followed by administration of the SCAT3 and ImPACT. About half the participants (but no more than 20 at a time) first took the ImPACT test in its entirety in a computer lab with two monitors present. The other half first met individually with an examiner who administered the SCAT3 in its entirety. Following the completion of this stage of testing, the participants exchanged testing methods. Of the 22 symptoms assessed through structured interview using the SCAT3, there were 18 symptoms comparable to the symptoms assessed on the ImPACT self-report. The descriptions contained exact or highly similar words and phrases for each symptom. The list of symptoms used from each measure appears in Table 1.
List of Symptoms Used From SCAT3 and ImPACT Measures to Comprise the 18 Symptoms Assessed
| SCAT3 | ImPACT |
|---|---|
| Headache | Headache |
| Nausea or vomiting | Vomiting |
| Dizziness | Dizziness |
| Blurred vision | Visual problems |
| Balance problems | Balance problems |
| Sensitivity to light | Sensitivity to light |
| Sensitivity to noise | Sensitivity to noise |
| Feeling slowed down | Feeling slowed down |
| Feeling like “in a fog” | Feeling mentally foggy |
| Difficulty concentrating | Difficulty concentrating |
| Difficulty remembering | Difficulty remembering |
| Fatigue or low energy | Fatigue |
| Drowsiness | Drowsiness |
| Trouble falling asleep | Trouble falling asleep |
| More emotional | Feeling more emotional |
| Irritability | Irritability |
| Sadness | Sadness |
| Nervous or Anxious | Nervousness |
| “Pressure in head”* | Nausea* |
| Neck pain* | Sleeping more than usual* |
| “Don’t feel right”* | Sleeping less than usual* |
| Confusion* | Numbness or tingling* |
Abbreviations: ImPACT = Immediate Post-Concussion Assessment and Cognitive Test; SCAT3 = Sport Concussion Assessment Tool–3rd Edition.
*No paired match.
The institutional review board of Florida Institute of Technology approved the study.
Statistical Analyses
The data were analyzed using the IBM Statistical Package for the Social Sciences (SPSS; IBM, Armonk, NY) version 24. One concern with the symptom data obtained in both settings was the large number of “no” (zero score) responses for individual symptom reports, as well as for the entire scale. These were not considered to be “excess” or “inflated” zeros since they reflected the actual status report of the participant. However, the statistical challenges presented by having many zero scores is formidable. We used nonparametric, Wilcoxon signed-rank tests for comparison of the structured interview and computerized self-report symptoms due to the skewed nature of the symptom endorsement and did not employ special tests that involve zero inflation.
Results
Participants reported significantly more symptoms (p < .001) in the face-to-face structured interview using the SCAT3 items (M = 1.44; SD = 1.97) compared to the computerized self-report using the ImPACT symptom scale (M = 1.26; SD = 2.36). Mean symptom severity endorsement (0–6 scale) between the structured interview (M = 2.19; SD = 3.72) and computerized self-report (M = 2.32; SD = 5.04) was not significantly different (p > .05). (See Table 2 for means and standard deviations.)
Means and SDs for the Overall Symptom Frequency and Severity of SCAT3 and ImPACT
| Measure | SCAT3 | ImPACT |
|---|---|---|
| Symptom frequency | M = 1.44*; SD = 1.97 | M = 1.26; SD = 2.36 |
| Symptom severity | M = 2.19; SD = 3.72 | M = 2.32; SD = 5.04 |
Abbreviations: ImPACT = Immediate Post-Concussion Assessment and Cognitive Test; SCAT3 = Sport Concussion Assessment Tool–3rd Edition.
*p < .001.
While the significant difference in total symptom endorsement between the two reporting modalities appears to represent a rather small absolute difference, the breakdown of these differences to frequency reports provides more cogent information. As shown in Figure 2, which shows frequency of symptom report, the most interesting datum is the difference in how many athletes reported zero symptoms. Significantly more individuals (421) reported zero symptoms on the ImPACT compared to that of individuals endorsing zero symptoms (336) on the SCAT (p < .001).
—Comparison of individual symptom endorsement. ImPACT = Immediate Post-Concussion Assessment and Cognitive Test; SCAT = Sport Concussion Assessment Tool.
Citation: International Journal of Athletic Therapy and Training 24, 2; 10.1123/ijatt.2018-0028
Individual Symptom Comparisons
A symptom-by-symptom portrayal of the individual symptoms for each report modality is shown graphically in Figure 3. The figure shows the percent endorsement of symptoms on the structured interview and the computerized self-report scales. Instead of consistency of report across the two formats in the same session, athletes differed significantly on several items. Symptoms reported significantly more often during the structured interview versus the computerized self-report scales were feeling slowed down, difficulty concentrating, difficulty remembering, fatigue, drowsiness, and feeling nervous/anxiety. Symptoms reported significantly more for ImPACT included sadness, more emotional, difficulty sleeping, light sensitivity, dizziness, and nausea.
—The percent endorsement of symptoms on the structured interview via SCAT3 and the computerized self-report scales via ImPACT. ImPACT = Immediate Post-Concussion Assessment and Cognitive Test; SCAT3 = Sport Concussion Assessment Tool–3rd Edition.
Citation: International Journal of Athletic Therapy and Training 24, 2; 10.1123/ijatt.2018-0028
Since the feeling of fatigue was such a likely endorsement on the SCAT3 interview, the possible role of that symptom in controlling the overall difference in symptom reports was assessed by removing it from the analysis. When fatigue was removed from both structured interview and computerized self-report, the frequency of symptoms on structured interview (p = .05; M = 1.15, SD = 1.73) remained higher than computerized self-report (M = 1.13, SD = 2.21). Severity of symptoms was not significantly different (p = .69).
Sex Comparisons
As shown in Table 3, females always reported more symptoms than did males on the structured interview (p = .01) and on the computerized self-report (p < .001), and more symptom severity on the structured interview (p = .003) and on the computerized self-report measures (p < .001). The modality of testing did not differentiate females’ symptom or severity reports. With males, significantly more symptoms, and at greater severity, occurred on the structured interview compared to the computerized self-report symptom scale (p < .001).
Means and SDs for the Overall Symptom Frequency and Severity of SCAT3 and ImPACT by Sex of Athlete
| Measure | SCAT3 | ImPACT |
|---|---|---|
| Symptom frequency | ||
| Males | M = 1.29; SD = 1.81 | M = 1.02; SD = 2.25 |
| Females | M = 1.67*; SD = 2.18 | M = 1.64**; SD = 2.48 |
| Symptom severity | ||
| Males | M = 1.86; SD = 3.09 | M = 1.75; SD = 4.19 |
| Females | M = 2.73*; SD = 4.49 | M = 3.23**; SD = 6.06 |
Abbreviations: ImPACT = Immediate Post-Concussion Assessment and Cognitive Test; SCAT3 = Sport Concussion Assessment Tool–3rd Edition.
*p ≤ .01, **p < .001.
Examiner Sex
Sex of the examiner who conducted the structured symptom interview did not result in any differences in reports of total symptoms or symptom severity for male or female participants. Although there were many more subjects tested by female examiners, a cross tabulation of examiner sex by participant sex showed no difference in expectancy of occurrence for any of the four combinations of female versus male examiners testing females and males (p = .903).
Order of Administration
The order in which participants were administered the structured interview and computerized self-report significantly differentiated symptom total and severity for the modalities. When the SCAT3 was administered first, athletes endorsed significantly more symptoms than on the ImPACT self-report (M = 1.23, SD = 2.15; p < .001). Similarly, greater severity on the structured interview (M = 2.47, SD = 3.91) compared to computerized self-report was found (M = 2.14, SD = 4.39; p = .001).
Discussion
Using a repeated-measures design, the current study compared symptom endorsements at the preparticipation baseline during a face-to-face structured interview using the symptom items of the SCAT3 versus the comparable symptom items of the computerized ImPACT test. Athletes reported a significantly greater number of symptoms in the interview format of the SCAT3 compared to the ImPACT self-report. Symptom severity endorsement (0–6 scale) between the structured interview and computerized self-report was not significantly different. Previous research has shown that environmental distractions increase symptom reporting during group administration of ImPACT.19 To the extent that any such distractions were present in the current study, the difference between interview versus ImPACT self-report might have been greater without distraction.
Previous studies that have compared baseline symptom reporting across test modalities have all employed group designs, where within-subject error variance is enhamced.10 The within-subject comparisons used in the current study provide a stronger clinical statement that individuals report more symptoms in the face-to-face interview process. These findings are dissimilar from the prior research examining baseline symptom reporting. Specifically, using a cross-sectional design, Krol et al10 found that athletes who completed the scale independently (by self-report) endorsed a significantly greater number of symptoms and a significantly greater symptom severity score than a set of athletes whose symptom information was obtained via interview.
As noted in the results, although the total symptom and symptom severity comparisons between the structured interview and computerized self-report resulted in significant differences, the magnitude of the differences was small (less than one item). Although it could be argued that a difference in 1-point at baseline on the symptom scale is more meaningful than a 1-point difference on the severity scale, neither may reach the level of clinical significance. More noteworthy, we think, was the difference in reporting between test formats for athletes who reported no symptoms. While zero was the modal and median score for symptom report regardless of test format, finding that nearly 10% more reported zero with the ImPACT self-report suggests something more is going on. Quite possibly, the structured interview with the examiner prompted more thought about the symptom presence, whereas it was easier to mark zero symptoms on the computer self-report.
Individual Symptom Comparisons
Of the 18 symptoms compared, six symptoms (e.g., feeling nervous/anxious, drowsiness, fatigue, difficulty remembering, difficulty concentrating, and feeling slowed down) were endorsed significantly more often by athletes responding in the structured interview versus self-report. Since the fatigue item was such a likely endorsement on the structured interview, the possible role of this symptom in controlling the overall difference in symptom endorsement was assessed by removing that symptom from the analysis. Even when fatigue was removed, the frequency of symptoms on the structured interview remained significantly greater than computerized self-report.
Sex Comparisons
Females reported significantly more symptoms and at greater severity on the structured interview and on computerized self-report measures than males. These findings are consistent with much previous research that examined symptom reporting where young female athletes reported physical symptoms nearly twice as often as their male counterparts.7,20,21 However, when we compared reporting on the two formats of symptom scales, it was only males who differed between interview versus computerized self-report. Specifically, males reported a significantly greater number of symptoms and greater symptom severity endorsement in the structured interview compared to the computerized self-report symptom scale. Such findings are not common in the literature on sex differences, where men seem more likely to report fewer experiences of pain than women, when interviewed by women.22 Perhaps a more parsimonious explanation is that the women maintained a higher level of engagement in testing regardless of method used or examiner sex than did the men.
Examiner Role
The present research examined demand characteristics, such as the sex of examiner on rate of responding. It was reassuring that examiner sex did not moderate any differences in scores on the structured interview for male and female athletes.
Order of Administration
When the SCAT3 was administered to participants before the ImPACT measure, athletes reported significantly more symptoms and at a greater severity on the structured interview than the computerized self-report. No significant differences were found if the ImPACT assessment was administered first. This finding is interesting and may be due to the initial presence of the interviewer causing greater sensitivity to the athletes in the SCAT first session, rather than the computerized introduction in ImPACT for those who took the ImPACT test first.
Limitations
The limitations for the present study include the inability to control for the presence of examiners, which was confounded across both types of administration. That is, no self-report completed in private was assessed. Additionally, we chose two forms to compare, a structured interview and a computerized self-report measure. A paper and pencil self-report measure was not utilized.
Clinical Implications
This study has implications that may assist clinicians in concussion management programs in recognizing that differences in symptom reporting may be expected based only upon the method/format of eliciting the symptom reports. We provide empirical support for using the same format at baseline, posttrauma evaluations, and follow-ups.
Future Research
The present study argues for uniformity across symptom collection methods, whether they are structured interviews, a computerized self-report, or nonsupervised paper and pencil self-reporting. Future research should aim to determine empirically if one or more of these measures is superior to the other in better capturing the presence or absence of a symptom.
Conclusions
The present study had a robust sample size and used standard test materials in concussion management, including ImPACT, the most widely used computerized measure in concussion.3 The symptom scale of the SCAT3 and computerized self-report symptom scale of the ImPACT both are derivatives of the Pittsburgh Post-Concussion Scale.18 The study has implications that may assist concussion management programs in recognizing differences of symptom reporting between the baseline mode of administration. Although we cannot conclude that one way of gathering symptom information is better than the other, since no objective, convergent, veridical symptom measure is available, the critical take-away message is that consistency in format of symptom report is imperative across testing sessions.
References
- 8.
Shehata N, Wiley JP, Richea S, et al. Sport concussion assessment tool: baseline values for varsity collision sport athletes. Br J Sports Med. 2009;43:730–734. PubMed ID: 19460765 doi:10.1136/bjsm.2009.059832
- 19.↑
Schatz P, Neidzwski K, Scolaro Moser R, Karpf R. Relationship between subjective test feedback provided by high-school athletes during computer-based assessment of baseline cognitive functioning and self-reported symptoms. Arch Clin Neuropsychol. 2010;25(4):285–292. PubMed ID: 20363733 doi:10.1093/arclin/acq022
- 21.↑
Eiser C, Havermans T, Eiser J. The emergence during adolescence of gender differences in symptom reporting. J Adolescence. 1995;18(3):307–316. doi:10.1006/jado.1995.1021
