Attentional focus is the process by which an individual devotes mental energy to cues, stimuli, or situations in relation to their performance in sport, exercise, or within an experimental setting (Neumann, 2019). A commonly investigated contrast is to have participants adopt either an internal or external focus. An internal focus refers to instructions or feedback directed at the body movements being performed; an external focus directs attention to the effects of those body movements on the environment (Wulf, 2013; Wulf et al., 1998). In the first paper on this topic, Wulf et al. (1998) demonstrated that the use of external cues led to superior performance and learning relative to internal cues. Systematic reviews have subsequently concluded that adopting an external focus can improve both performance and learning in a variety of task categories with participants of diverse expertise, skill, and age profiles (Chua et al., 2021; Wulf, 2013). Such conclusions are predicated on the contributing studies following sound methodological procedures (Ranganathan et al., 2022), such as the inclusion of manipulation checks (Hoewe, 2017).
Festinger (1953) advised that “It is rarely safe to assume beforehand that the operations used to manipulate variables will be successful and will tie in directly with the concept the experimenter has in mind. It is a worthwhile precaution to check on the success of the experimental manipulations” (p. 145). A manipulation check is a test that evaluates the efficacy of an experimental design manipulation (Hoewe, 2017); that is, did the participants adhere to the instructions that they received. In mainstream psychology, manipulation checks can provide important evidence in the evaluation of concepts. For example, in a meta-analysis on the effectiveness of growth mindset interventions, Macnamara and Burgoyne (2022) reported a concerning finding that “among the studies that reported manipulation checks, the meta-analytic effect on academic achievement was significant for studies where the manipulation checks failed but nonsignificant for studies where the manipulation checks succeeded” (p. 3), a confusing result if growth mindset interventions are indeed effective. Manipulation checks may offer similar assistance in evaluating the efficacy of concepts in motor learning (Ranganathan et al., 2022).
Initial research conducted by Wulf et al. (1998, 1999, 2000) on attentional focus did not include any manipulation check. Early research on the topic identified a lack of adherence to instructions by some participants (Maxwell & Masters, 2002), while Mullen (2007) identified the absence of manipulation checks as a “fundamental problem with the attentional focus research” (p. 40). More recently, diverse manipulation checks have been used within focus of attention research (see Table S1 in Supplementary Material [available online]) consistent with guidance on quality experimental design within motor learning (Ranganathan et al., 2022). Yamada, Raisbeck, and Porter (2021) reviewed participants’ adherence to attentional focus instructions, reporting that the data collection methods varied from simple yes/no questions (Stoate & Wulf, 2011) to closed questions using a Likert-type (Bell & Hardy, 2009) or 0%–100% scale (MacMahon et al., 2014), or employed open-ended questions (Porter et al., 2010). Yamada et al. further reported that many studies adopted a combination of two or more methods (e.g., Becker et al., 2020) and that participants generally adhered to their attentional focus instructions.
While the majority of manipulation checks have relied upon self-report from participants, Yeh et al. (2016) designed an objective manipulation check. Participants were asked to balance on a force platform under either external or internal focus conditions with visual feedback present in the first half and absent in the second half of each trial. Postural control was compared with visual feedback present and absent; due to the specific instructions provided, adhering to an external focus would produce postural control changes along the mediolateral but not anterior–posterior direction when visual feedback was withdrawn. In contrast, adherence to the internal focus instructions would not produce any difference in postural control when feedback was present or absent. The authors concluded by recommending the use of such objective manipulation checks in future studies (see also Ejelöv & Luke, 2020; Fiedler et al., 2021; Hauser et al., 2018), but the extent to which their advice has been followed is unknown. Indeed, there is limited detail available on the prevalence of manipulation check use more generally; a recent systematic review on the use of attentional focus instructions in strength training highlighted that only two of 15 studies reviewed included a manipulation check (Pompa et al., 2024). While the diversity of manipulation checks in use within focus of attention research has been acknowledged, the general prevalence of their use and the extent to which higher quality manipulation check protocols are employed has not been systematically investigated.
The effective use of manipulation checks can strengthen a reader’s confidence in the results arising from an experiment (Macnamara & Burgoyne, 2022; Markwell et al., 2022), especially when contrasted against experiments that either omit the use of manipulation checks (Wulf et al., 1998) or use checks of questionable efficacy (Stoate & Wulf, 2011). However, this confidence requires not just that the manipulation check be conducted but that it be acted upon within the processing and/or analysis of data (Aiken & Becker, 2023). Manipulation checks may be failed for several reasons, such as unclear instructions, participant noncompliance, or inadvertent cues affecting participant behavior (Hoewe, 2017; Ranganathan et al., 2022). The inclusion of misleading data arising from failed manipulation checks could potentially lead to flawed findings or skewed results, leading to inappropriate guidelines for practitioners. Consider the example of a pre/post design testing the efficacy of a nutritional supplement. After the posttest, a participant in the experimental group returns all of the packets of the supplement unopened, explaining that they chose not to take them after all. This participant would be excluded from the analysis. Similar examples of manipulation check failures can be found in the attentional focus literature (Couvillion & Fairbrother, 2018; Freudenheim et al., 2010; Markwell et al., 2022; McNicholas & Comyns, 2020), although the approach taken to dealing with these manipulation check failures is highly variable. For example, McNicholas and Comyns (2020) analyzed all participants in their assigned conditions, irrespective of manipulation check results. Freudenheim et al. (2010) excluded those participants who failed the manipulation check from further analysis, whereas Markwell et al. (2022) performed a second analysis with noncompliant participants removed to determine if compliance led to significant changes. The impact of varying analysis protocols for dealing with manipulation check failures has not been systematically evaluated.
The primary aim of this methodological systematic review was to clarify how often manipulation checks are used, the types of manipulation checks used, and how manipulation checks were analyzed in research on focus of attention. To offer an appraisal of contemporary practice, the review was restricted to articles published in the 2 years prior to the start of the project. In addition, a secondary data analysis of selected studies was used to evaluate the impact of manipulation check analysis strategies on study outcomes. The results from this review should stimulate discussion in relation to methodological best practice so that future research on focus of attention will provide more dependable results across a range of different domains including sport and rehabilitation.
Method
Research Design
This methodological systematic review was conducted according to the general guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (Page et al., 2021; see Table S3 in Supplementary Material [available online]). The review was registered on the Open Science Framework (https://osf.io/69dpx/) to facilitate the transparency of the review (Foster & Deardorff, 2017). Minor changes to the registered protocol were made during the research (e.g., additional variables coded within the extraction table and refining of coding options for some questions); both the original and final extraction tables are available on the Open Science Framework project page.
Procedure
The following internet databases were searched for possible relevant research that looked at the impact of attentional focus on motor learning and performance: APA PsycINFO, PubMed, SPORTDiscus, and Web of Science. The following terms and Boolean operators were used for the search based on previous reviews in this area of research: (“focus of attention” OR “attentional focus” OR “attentional focusing”) AND (“internal focus” OR “external focus”) AND (“performance” OR “learning” OR “instruction”). The search was refined by date of publication to include research from September 2021 to September 2023.
After results were obtained, a search validation procedure (Booth, 2010) was carried out to ensure each researcher conducted the same search and that results were consistent, and accurate across all the databases included in the review. Initially, two researchers (Curran and Kearney) independently completed the initial search and confirmed the same number of results from each database. Subsequently, a Google Scholar search was performed using the search terms “attentional focus” AND “external focus” AND “performance” OR “learning” to make sure the most influential papers in the area (as determined by the search algorithm) were present in the search findings. The resulting papers were reviewed in order of appearance for appropriateness using our inclusion criteria, noted below. The first 20 relevant papers were selected. All 20 papers were returned within our search of the databases, validating the search procedure in this systematic review.
All articles were uploaded to a Rayyan (https://www.rayyan.ai/) database where they could be screened. Inclusion criteria were that the paper reported an intervention or experiment with one or more comparisons between an external focus, internal focus, or control group; participants of any age, gender, health status, or experience level; and published in the English language within the 2 years prior to the start of the project to capture contemporary practice. Observational studies and reviews were excluded. The automated duplicate removal tool in Rayyan was used to remove any duplicate studies found in the initial search.
An initial round of screening focused on the title and abstracts of the retrieved records. The screening process was carried out by two researchers (Curran and Kearney) independently, and their respective opinions compared to ensure an accurate screening procedure. Uncertainty in relation to the inclusion of specific studies was resolved through discussion with a third author (Nugent), and a consensus was reached on all research articles included or excluded. A second round of screening focused on the full texts of the articles selected in the initial round. When the screening process was complete, a total of 78 protocols from 76 articles were included in the systematic review. Details of records identified, screened, excluded, and so forth, can be found in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart below (Figure 1).
—PRISMA flowchart of the search procedure. PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Citation: Journal of Motor Learning and Development 13, 1; 10.1123/jmld.2024-0069
Data Extraction and Synthesis
Two researchers (Curran and Kearney) independently extracted the required information from all papers. Information from each paper was added to an extraction table using Microsoft Excel where findings could be compiled and analyzed. Where a paper reported two experiments, if the manipulation check protocol was the same across the experiments, then only one protocol was coded. If the experiments involved different protocols, each experiment was coded. Where different groups experienced different manipulation check protocols (e.g., open questions for a “no instruction” group and closed questions for groups which received specific instructions), both protocols were coded. Where the same protocol and data set (i.e., same participants, same data collection) was reported in two different papers examining different outcome variables, only one article was coded.
The questions in the extraction table included: Authors, Year of Publication, Sample Size, Age, Gender, Experience Level, Groups/Conditions (e.g., Internal, External, and Control), Study Design (repeated measures or independent groups), and Exercise/Movement Completed (e.g., dart throw). In relation to the collection of Manipulation Check data, information was extracted in relation to the type of Manipulation Check Used, the Precise Manipulation Check (quoted from each paper), When Manipulation Check was used, whether the Manipulation Check Results were Reported, and what the Manipulation Check Results were (quoted from paper). In relation to the subsequent analysis of the manipulation check undertaken in each experiment, information was extracted on whether the Manipulation Check Failures were Analyzed and the Reason not Analyzed (specific details from each paper, if provided). Finally, to capture broader factors which might influence the effectiveness of a manipulation check, we coded whether participants were Familiarized with Focusing Instructions and whether Reminder(s) of Instructions were Provided. A link to the full extraction table containing both the response options for each of these questions and the extracted data can be found here: https://osf.io/69dpx/.
As the focus of this methodological review was on the procedures used within the literature, a risk of bias assessment was not completed for individual studies (Waffenschmidt et al., 2019). A customized Microsoft Excel spreadsheet was used to calculate the number and percentage of responses for each column in the extraction table.
Secondary Data Analysis
To analyze the effect of including manipulation check failures in the analysis on the outcomes of papers, a secondary data analysis was conducted. Eighteen papers were deemed suitable for secondary data analysis on the basis of (a) comparing performance on a motor task under two or more attentional focus conditions, (b) including a manipulation check, and (c) the authors not having completed additional analyses on the basis of manipulation check failures. No reply, or an initial reply but no data, was received from eight authors who were contacted by email. For 10 papers, data were available via an online repository or received following communication with the authors. For one of the received data sets, the performance score was a complex calculation from several covariates produced at mean group level rather than as an individual score; consequently, this data were not suitable for our reanalysis. For two more of the received data sets, no participants failed the manipulation check. Of the seven remaining studies, two studies had two primary variables, and another study analyzed both skilled and novice participants; this resulted in 10 data sets for reanalysis. As the focus of this article is on the outcome of the analysis protocols used rather than on the specific studies, the seven specific papers reanalyzed have not been named.
For studies that used more than one measure to assess the impact of focus of attention on motor performance or its neuromuscular correlates, we reanalyzed the primary outcome variable relating to performance (e.g., distance thrown and error from the target). Where there were multiple primary outcome variables, all primary variables were reanalyzed.
For each available data set, the analysis performed by the original authors on the full data set was initially repeated. In one case, it was not possible to reproduce the original analysis. An independent statistician received the raw data and advised on an appropriate analysis procedure. Implementing this procedure produced the same outcome as the original analysis.1 A second analysis was then conducted analyzing only the participants who passed the manipulation check. A “pass” was defined as reaching a certain criterion or score based on each data set. For papers that used a rating scale (e.g., 0–10 and 1–7), participant scores were converted to a percentage, and a score equivalent to 60% or more was chosen as it was the most commonly applied in previous research on attentional focus (Richer et al., 2017; Sawai et al., 2022). For papers that used an agreement scale from Strongly Agree to Strongly Disagree, the pass criterion was set as at least “Agreed.” For studies in which participants reported what percentage of their attention was allocated to each of a number of foci (e.g., internal, external, and other), participants passed the manipulation check if the intended focus instructed to them was the highest out of the foci reported. For repeated measures designs, participants were required to have passed all manipulation checks on all conditions to be included in the secondary analysis.
The original analysis (e.g., paired t test and analysis of variance) used by the authors of these papers was repeated unless there was an assumption for the particular test that was violated by the smaller sample size when the manipulation check failures were removed. In such cases, the nonparametric equivalent tests were performed (e.g., Wilcoxon signed-rank test for paired t test). To investigate the effect of removing manipulation check failures on sample size, a Wilcoxon signed-rank test compared the sample sizes for the original and reanalysis data sets.
Results
Methodological Systematic Review
Figure 1 illustrates the process of arriving at the final sample of 78 protocols from 76 articles. The initial search returned a total of 452 articles: 204 articles from SPORTDiscus, 35 articles from APA PsycINFO, 82 from PubMed, and 121 articles from Web of Science. Once duplicate papers were removed from the database search results, the total was reduced to 320. By screening titles and abstracts relative to our inclusion criteria, a further 225 papers were removed. There were 95 records left for full text review; at this stage, another 12 papers were removed from the study for reasons such as adopting a nonexperimental approach. Full texts were not available for a further seven papers. The characteristics of the final 78 manipulation check protocols included in this review can be observed in Table S2 (see Supplementary Material [available online]).
Of the 78 protocols included in this systematic review, 41 (53%) had no manipulation check (Table 1). In 22% of experiments without a manipulation check, a reminder of the instructions was provided before every trial (e.g., Furuhashi et al., 2022; Slovák et al., 2023; Woodard et al., 2021); in several additional experiments, participants were asked to repeat the instructed focus prior to the start of a block of trials to confirm understanding (e.g., Mazza et al., 2022; Siltanen & Bottas, 2022). Of the 21 papers using a repeated measures design that conducted manipulation checks, 19 (90%) conducted the manipulation checks during or after each specific focus condition to minimize recall errors.
Summary of Extraction Table Results
| Question | N (%) | Response options |
|---|---|---|
| Were participants explicitly given practice trials with the cues (alongside or distinct from practice with the task)? | 67 (87%) | No |
| 19 (13%) | Yes | |
| Was a manipulation check used (and if so, what type?)? | 41 (53%) | No manipulation check |
| 15 (19%) | Single rating scale (e.g., How much did you focus on X? 1%–5 and 0%–100%) | |
| 9 (11.5%) | Multiple rating scale questions (e.g., how much did you focus on X, how much on Y? etc.) | |
| 9 (11.5%) | Open questions | |
| 3 (4%) | Forced choice question (Did you focus on A, B, or C? Answer Y/N to each of these) | |
| 1 (1%) | Behavioral measures specific to the instruction | |
| 0 (0%) | Yes/No question (Did you focus on X?) | |
| When was the manipulation check used (repeated measures designs)? | 16 (76%) | After each condition |
| 3 (14%) | During each condition | |
| 2 (10%) | After all conditions | |
| 0 (0%) | Not stated | |
| Were reminders of the instructions provided? | 57 (74%) | Yes |
| 20 (26%) | No | |
| Were the results of the manipulation check reported in the results (or method)? | 24 (65%) | Yes, average results only |
| 6 (16%) | Yes, average and failures | |
| 5 (14%) | Yes, number of failures only | |
| 2 (5%) | No | |
| What did the researchers do with participants who failed the manipulation check? | 25 (68%) | The authors did not specify whether anyone failed or not |
| 4 (11%) | Failures reported and included | |
| 3 (8%) | Failures reported and excluded | |
| 3 (8%) | Two analyses run | |
| 2 (5%) | The authors specified that no participant failed the manipulation check |
In terms of the type of manipulation checks used, 15 studies (19%) asked one single rating question to participants as a manipulation check. In nine protocols (11.5%), multiple questions about adherence to the task manipulation checks were asked while an open question as to what participants focused on was the manipulation check of choice for an additional nine protocols (11.5%) in the review. While the majority of Open Questions asked a variation of “what did you focus on?” (e.g., Allingham et al., 2021; Ghanati et al., 2022; Higgins et al., 2021), one paper simply asked participants to recall the instructions provided (Yamada et al., 2022). While still an open question and revealing obvious manipulation check failures, the latter example may not pick up all failures (i.e., those who could remember but did not employ the instructions). Only one paper used a behavioral manipulation check; Sakurada et al. (2022) included a manipulation whereby if participants adhered to the instructions, they would produce a specific pattern of reaction time results. Four papers used two-part manipulation checks, where an initial question asked participants to identify the instruction they had been provided with, and a second question asked those who could accurately report the instruction how much they had focused on this instruction (Aiken & Becker, 2023; Yamada & Raisbeck, 2021; Zhuravleva & Aiken, 2023; Zhuravleva et al., 2023).
The majority of papers that included a manipulation check reported the results of those manipulation checks: 65% reported average results per group or condition, 16% reported both averages and the number of manipulation check failures, and 14% reported the number of manipulation check failures only. The remaining 5% did not report any manipulation check results.
In terms of analysis, three studies (8%) ran two analyses (Aiken & Becker, 2023; Higgins et al., 2021; Markwell et al., 2022): one including all participants and the second including only those who failed the manipulation check. Three further studies excluded manipulation check failures from the analysis (Lamers James & O’Connor, 2023; Neumann et al., 2022; Sawai et al., 2022); however, while Neumann et al. (2022) reported that “Three participants did not adhere to the attentional focus instructions and were excluded” (p. 2), they did not define what response was required to pass the manipulation check. The remaining studies either explicitly included manipulation check failures in the analysis (11%; Allingham et al., 2021; Chen et al., 2021, 2022; Howard et al., 2023), specified that no participants failed the manipulation check (5%; Ma et al., 2022; Yamada et al., 2022), or did not specify whether anyone had failed or not (68%).
While 77% of protocols included familiarization trials with the task to be performed, only 13% of papers provided participants with additional familiarization trials with the cues to be used within data collection trials (e.g., de Arruda et al., 2024; Neumann et al., 2022; Vargas-Molina et al., 2022). Fifty-seven studies (74%) provided participants with reminders of instructions at some stage during the experiment.
Reanalysis of Selected Data sets
The Wilcoxon Signed Ranks test revealed that removal of participants who failed the manipulation check led to a significant reduction in the sample size from the original (Mdn = 17, interquartile range = 11) to the reanalysis (Mdn = 13, interquartile range = 2), z = −2.527, r = −.89, p = .012. Eight of the 10 data sets reanalyzed showed consistent results when the analysis was repeated with only those participants who passed the manipulation check (Table 2). Reanalysis of Secondary Data Analysis paper 3 (SDA3) showed a drop in participant numbers by two, and the results of the paired t test comparing the internal and external conditions changed from p = .037 to p = .131. Reanalysis of paper SDA4 found that, in comparison to the significant one-way analysis of variance indicating a difference in performance of the three groups in the original paper (p ≤ .001), the reanalysis of the much smaller sample of participants who passed all manipulation checks showed no significant difference between conditions (p = .084).
Results of Original and Reanalysis of Data Sets Included in the Review
| Paper | Comparison | NO | NR | Result of reanalysis |
|---|---|---|---|---|
| SDA1 | I vs. E vs. H | 18 | 14 | No change in outcome. |
| SDA2 | I vs. E vs. H | 16 | 12 | No change in outcome. |
| SDA3 | I vs. E | 15 | 13 | Significant difference in original analysis; no significant difference in analysis of those who passed MC only. |
| SDA4 | I vs. E vs. Combination | 27 | 13 | Significant difference in original analysis; no significant difference in analysis of those who passed MC only. |
| SDA5a (simple RT) | I vs. E | 19 | 16 | No change in outcome. |
| SDA5b (choice RT) | I vs. E | 19 | 16 | No change in outcome. |
| SDA6a (force) | I vs. E | 27 | 14 | No change in outcome. |
| SDA6b (success) | I vs. E | 27 | 14 | No change in outcome. |
| SDA7a (skilled) | I vs. E vs. H | 14 | 8 | No change in outcome. |
| SDA7b (novice) | I vs. E vs. H | 14 | 12 | No change in outcome. |
Note. NO = number of participants in the original analysis; NR = number of participants in the reanalysis with participants who failed the manipulation check excluded; MC = manipulation check; I = internal focus condition; E = external focus condition; H = holistic focus condition; SDA = secondary data analysis.2
Discussion
Manipulation checks are an important element of quality experimental design in motor learning (Hoewe, 2017; Ranganathan et al., 2022). This systematic methodological review identified a range of considerations for scientists working on focus of attention research to reflect upon. Just under half of papers included a manipulation check. Where manipulation checks were included, they were typically of good quality. However, there was a lack of transparency in terms of reporting whether any participants failed the manipulation check, and in terms of how such cases were treated during data analysis. For 10 data sets, participants who had failed the manipulation check were removed and the analysis was then repeated. In two of these data sets, significant differences between attentional focus conditions from the original analysis were not significant on reanalysis.
Just over 50% of reviewed studies had no manipulation check. Verbal manipulation checks are a straightforward addition to experimental protocols, which help to rule out alternative explanations for the observed results (Ranganathan et al., 2022). The dominant concern with the use of manipulation checks is reactivity: that the timing and content of the manipulation check would change the behavior of the participant (Ejelöv & Luke, 2020; Fayant et al., 2017; Hauser et al., 2018; Ranganathan et al., 2022). Within the context of focus of attention experiments, however, it is likely that the benefits to be gained from utilizing manipulation checks outweigh any potential reactivity. One solution to this concern is for the manipulation check to be an incidental behavioral measure, rather than relying on self-report (Ejelöv & Luke, 2020; Fiedler et al., 2021; Hauser et al., 2018; Yeh et al., 2016). A clear example of a behavioral manipulation check was provided by Toner and Moran (2011). Golfers were tasked with making specific changes to the path that their putter took; the SAM Putt Lab system was used to track the extent to which golfers were able to make the desired changes. Among the reviewed studies, a similar approach was adopted by Sakurada et al. (2022). One important criticism of behavioral manipulation checks is that they measure the desired effect of the instruction rather than the adherence per se.3 That is, a golfer might devote 100% of their attention to trying to produce a change in the movement but fail and produce no behavioral change at all. Conversely, a golfer could be instructed to focus on the movement of their golf club but decide they are going to focus on their hands instead; since the club and hands are coupled, the desired change in club kinematics could occur—suggesting successful adherence to the instruction—even when the golfer thought about something else entirely. Thus, where behavioral manipulation checks are used, it may be appropriate to supplement them with self-reports. Furthermore, given the wide range of applications in which focus of attention instructions are employed (Chua et al., 2021; Wulf, 2013) and the potential difficulty of designing a behavioral manipulation check for all conditions, it is likely that self-report manipulation checks will remain an important tool for the research scientist.
A second concern in relation to self-report manipulation checks is social desirability bias (Reynolds, 1982); that is, participants may respond in a way that they believe will satisfy the experimenter. The concerned experimenter can consider using the minimum necessary number of manipulation checks, the avoidance of leading questions, and emphasizing the importance of honest reporting in the preexperiment briefing (e.g., Markwell et al., 2023) to reduce the risk of social desirability. In addition, different members of the research team could collect performance and manipulation check data, further facilitating honest reporting. Critically, the precise protocol and rationale for the chosen manipulation check should be clearly described (Ejelöv & Luke, 2020; Hauser et al., 2018). If experimenters desire data in relation to social desirability bias, a short form questionnaire is available (Reynolds, 1982), which could be implemented at the end of the protocol (Kearney, 2015).
Manipulation check failures may be due to inadequate initial preparation of participants (Hoewe, 2017). In one of the studies subjected to secondary data analysis, only 13 of 27 participants successfully passed the manipulation checks for each of the three attentional focus conditions that they completed. While it is impossible to retrospectively identify why this was, a reasonable avenue to investigate is the preparation that participants received. For example, participants routinely receive familiarization with the tasks that they are to perform. Adding an additional familiarization block where participants have an opportunity to practice applying the instructional cues they are to utilize during data collection trials (Ranganathan et al., 2022), as demonstrated by several papers within the current review (Nutpiboonmongkol & Benjapalakorn, 2022; Sherman et al., 2021; Vargas-Molina et al., 2022) and encouraging participant questions to clarify and confirm their understanding during this practice (Aiken & Becker, 2023; Kearney, 2015; Lola et al., 2022), may reduce the likelihood of later manipulation check failures.
In addition to identifying an appropriate manipulation check, researchers should define what constitutes an acceptable response. As detailed in this review, only a minority of studies explicitly reported this information (Ma et al., 2022; Richer et al., 2017; Sawai et al., 2022). The criterion for successfully passing the manipulation check ranged from greater than 50% (Ma et al., 2022) to 60% (Aiken & Becker, 2023; Richer et al., 2017; Sawai et al., 2022) on one question checks or a simple majority of attention on more complex checks (Parr et al., 2023; Sakurada et al., 2022). It is beyond the bounds of this paper to suggest what an appropriate criterion should be; future research might consider the relationship between behavioral measures (Sakurada et al., 2022; Toner & Moran, 2011) and degree of adherence as a preliminary step toward answering this question. However, all papers should clearly identify the criterion to pass their manipulation check in advance (ideally as part of a preregistration) and fully report how many participants achieved this criterion.
The consequences of participants not passing the manipulation checks were illustrated by the reanalysis of data, where the results of two out of the 10 reanalyses led to a change in the interpretation of the results. Potential reasons for the change in results include the loss in statistical power due to the reduced sample size or a compliance effect where deviating from the set instructions could have masked the benefit, decrement, or null effects of different attentional strategies. The reanalysis demonstrated the importance of considering manipulation check failures; the strategy of ignoring manipulation check failures (Allingham et al., 2021; McNicholas & Comyns, 2020) seems untenable. Allingham et al. (2021) offered a counterargument to this position, suggesting that because manipulation check data is “limited in its ability to truly assess the degree to which a person focuses on a certain object, we do not use these data as a basis for exclusion or further analysis. Also, even though a participant might have exhibited difficulties following the focus instructions, they may have still been affected by the instructions at an implicit level” (p. 434). Allingham et al.’s concerns about the comprehensiveness of self-report is shared by other authors (e.g., Yamada, Higgins, & Raisbeck, 2021). It is certainly true that manipulation checks are as vulnerable to measurement issues, such as reliability, and validity, as any other measure (Fayant et al., 2017); thus, careful vetting of manipulation checks is an important step for researchers (Ejelöv & Luke, 2020). Rather than simply removing participants who fail a manipulation check from the analysis, researchers should explore reasons why these individuals did not respond as expected through a debriefing process (Hoewe, 2017); systematic reasons for a lack of compliance may provide valuable feedback to inform future study design.
Two other strategies were apparent in relation to handling manipulation check failures: repeating trials where the manipulation check results were below the threshold (Ma et al., 2022) and excluding manipulation check failures from further analysis (Lamers James & O’Connor, 2023; Sawai et al., 2022). The former strategy introduces an additional difference between participants, and thus is likely to be unsuitable in many cases. The latter strategy is the most conservative, with likely implications for sample size. Researchers planning future studies in which participants who fail the manipulation check are to be excluded from further analysis should consult similar studies to establish the likely failure rate and adjust the sample size estimation from their power analysis accordingly. In addition, there may be occasions when it is inappropriate to discard participants who failed a manipulation check (Aronow et al., 2019; Ejelöv & Luke, 2020). One such example is when evaluating the applicability of an intervention in a field setting. In such cases the variable of interest is whether the population has experienced a benefit and therefore restricting analysis to just those individuals who had successfully engaged with the intervention could overstate the efficacy of the intervention in that setting (e.g., physical education and club sport). However, when more basic research is being undertaken to establish the potential efficacy of a particular focus, the strategy of completing two distinct analyses, one including all participants and the second including only those who failed the manipulation check (Aiken & Becker, 2023; Markwell et al., 2022), should provide additional confidence in the robustness of results.
This methodological systematic review is limited to the 2-year period prior to the beginning of the study and to the published literature (i.e., no grey literature). As such, it may not tell the full picture of contemporary practice in relation to the use of manipulation checks in focus of attention research. However, the 2-year sample provides sufficient examples to stimulate reflection among researchers. A second limitation is that a risk of bias assessment of the individual studies was not included; omitting a risk of bias assessment is not unusual in methodological systematic reviews where the primary concern is with what methods were used rather than with the overall quality of experiments (Waffenschmidt et al., 2019). It may be that the studies that did not include a manipulation check were those that were of generally poorer quality. However, 68% of studies did not explicitly detail whether any participants had failed the manipulation check, and the results of our reanalysis of a subset of the available data reveal that clearly some participants did fail (see also Yamada, Higgins, & Raisbeck, 2021), with potential consequences for the conclusions drawn from each study.
In conclusion, the use of manipulation checks within focus of attention research is not consistent across sport, rehabilitation, and motor learning domains. Greater use of manipulation checks and greater transparency in analysis and reporting is required to increase confidence in the findings of future studies. This review can act a stimulus for researchers to consider how to more comprehensively conduct and report manipulation checks in their future experimental research. Specifically, in light of the above discussion, the following recommendations are made with respect to the use of manipulation checks in focus of attention research:
- •Researchers should use a manipulation check to check for compliance. If researchers deliberately choose not to use a manipulation check, then they should provide a clear and explicit reason for this choice within their method (e.g., pilot testing revealed that participants lacked the verbal skills to engage with the manipulation check; Krajenbrink et al., 2018).
- •To facilitate participant understanding and compliance during pilot testing, researchers should explore whether there is a need to provide participants with familiarization trials with the instructions, as well as familiarization trials with the task; in other words, to ensure that the manipulation check is sufficiently clear to minimize decisions to remove participants due to failed manipulation checks (Aronow et al., 2019).
- •Researchers should use a form of manipulation check that addresses the multiple possible foci that individual could adopt during the task; for example, in an experiment comparing the effects of internal and external foci, researchers could (a) ask participants how much they focused on each of the possible foci (e.g., Bell & Hardy, 2009), (b) ask participants to allocate a percentage to each focus they adopted during the practice trials (including the prescribed options and another category) with the total summing to 100% (Parr et al., 2023), or (c) ask an open question and qualitatively code the responses (Porter et al., 2010). Equally, experimenters might couple closed and open questions (Kearney, 2015).
- •Given concerns about self-report (Yamada, Higgins, & Raisbeck, 2021), social desirability bias (Kearney, 2015), and implicit effects of instruction (Allingham et al., 2021), research should further explore the potential for behavioral manipulation checks (Sakurada et al., 2022; Toner & Moran, 2011; Yeh et al., 2016), either alone or in conjunction with self-report measures.
- •In light of the call for increased standardization within motor learning protocols (Ranganathan et al., 2021), there would be merit in a Delphi study (Quartiroli, 2024) to propose an optimal form of manipulation check and protocol for implementing said check within focus of attention research.
- •Ahead of running an experiment, researchers should set a criterion for noncompliance with the instructions; that is, what score indicates a failed manipulation check? Researchers should explicitly report their criterion and the number of participants who failed the manipulation check.
- •When analyzing their data, researchers should either (a) exclude all individuals who failed a manipulation check from the subsequent analysis (Freudenheim et al., 2010), (b) run one analysis including all participants and a second analysis including just those who have passed the manipulation check (Aiken & Becker, 2023; Higgins et al., 2021; Markwell et al., 2022), or (c) provide explicit justification for the inclusion of all participants (Aronow et al., 2019).
Notes
This study showed no difference in outcome when the data were reanalyzed with manipulation check failures removed.
A holistic focus directs attention to the general feeling of the movement (e.g. smooth and bouncy) and is distinct from an internal focus, which directs attention toward a specific body part (Zhuravleva et al., 2023).
We would like to thank an anonymous reviewer for suggesting this caveat and example.
Acknowledgments
The authors would like to thank Dr. Masahiro Yamada and Prof. Edward Hebert for helpful discussions during the development of this manuscript.
References
Aiken, C.A., & Becker, K.A. (2023). Utilising an internal focus of attention during preparation and an external focus during execution may facilitate motor learning. European Journal of Sport Science, 23(2), 259–266.
Allingham, E., Burger, B., & Wöllner, C. (2021). Motor performance in violin bowing: Effects of attentional focus on acoustical, physiological and physical parameters of a sound-producing action. Journal of New Music Research, 50(5), 428–446.
Aronow, P.M., Baron, J., & Pinson, L. (2019). A note on dropping experimental subjects who fail a manipulation check. Political Analysis, 27(4), 572–589.
Becker, K.A., Fairbrother, J.T., & Couvillion, K.F. (2020). The effects of attentional focus in the preparation and execution of a standing long jump. Psychological Research, 84(2), 285–291.
Bell, J.J., & Hardy, J. (2009). Effects of attentional focus on skilled performance in golf. Journal of Applied Sport Psychology, 21, 163–177.
Booth, A. (2010). How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care, 26(4), 431–435.
Chen, C.-C., Ryuh, Y., Luczak, T., & Lamberth, J. (2021). The effects of attentional focus and skill level on the performance of golf putting. Journal of Motor Learning and Development, 9(3), 371–382.
Chen, C.-C., Ryuh, Y., Luczak, T., & Lamberth, J. (2022). Examining different foci of attention on golf putting performance in novice learners. Physical Educator, 79(4), 466–483.
Chua, L.-K., Jimenez-Diaz, J., Lewthwaite, R., Kim, T., & Wulf, G. (2021). Superiority of external attentional focus for motor performance and learning: Systematic reviews and meta-analyses. Psychological Bulletin, 147(6), 618–645.
Couvillion, K.F., & Fairbrother, J.T. (2018). Expert and novice performers respond differently to attentional focus cues for speed jump roping. Frontiers in Psychology, 9, Article 2370.
de Arruda, D.G., Dai, B., Readdy, T., McCrea, S., & Zhu, Q. (2024). Sequential focus of attention instructions influenced motor performance of volleyball setting despite direction of focus but dependent on motor expertise of the player. International Journal of Sport and Exercise Psychology, 22(1), 24–51.
Ejelöv, E., & Luke, T.J. (2020). “Rarely safe to assume”: Evaluating the use and interpretation of manipulation checks in experimental social psychology. Journal of Experimental Social Psychology, 87, Article 103937.
Fayant, M.P., Sigall, H., Lemonnier, A., Retsin, E., & Alexopoulos, T. (2017). On the limitations of manipulation checks: An obstacle toward cumulative science. International Review of Social Psychology, 30(1), 125–130.
Festinger, L. (1953). Laboratory experiments. In K.D. Festinger (Ed.), Research methods in the behavioral sciences (pp. 136–172). Rinehart & Winston.
Fiedler, K., McCaughey, L., & Prager, J. (2021). Quo vadis, methodology? The key role of manipulation checks for validity control and quality of science. Perspectives on Psychological Science, 16(4), 816–826.
Foster, E.D., & Deardorff, A. (2017). Open science framework (OSF). Journal of the Medical Library Association, 105(2), 203–206.
Freudenheim, A.M., Wulf, G., Madureira, F., Pasetto, S.C., & Correa, U.C. (2010). An external focus of attention results in greater swimming speed. International Journal of Sports Science & Coaching, 5(4), 533–542.
Furuhashi, Y., Hioki, Y., Yoshimoto, S., & Hayashi, R. (2022). Effect of neutral, internal, and external focus on drop jump performance: Is drop jump performance affected by plyometric training experience? The Journal of Strength & Conditioning Research, 36(12), 3319–3323.
Ghanati, H.A., Letafatkar, A., Almonroeder, T.G., & Rabiei, P. (2022). Examining the influence of attentional focus on the effects of a neuromuscular training program in male athletes. The Journal of Strength & Conditioning Research, 36(6), 1568–1575.
Hauser, D.J., Ellsworth, P.C., & Gonzalez, R. (2018). Are manipulation checks necessary? Frontiers in Psychology, 9, Article 998.
Higgins, L.Q., Labban, J.D., Stout, R.D., Fairbrother, J.T., Rhea, C.K., & Raisbeck, L.D. (2021). Postural control in older adults during and following a 12-week balance training intervention with attentional focus instructions. Journal of Motor Learning and Development, 9(3), 345–370.
Hoewe, J. (2017). Manipulation check. In J. Matthes, C.S. Davis, & R.F. Potter (Eds.), The international encyclopedia of communication research methods (pp. 1–5). John Wiley & Sons.
Howard, C.K., Van Gemmert, A.W.A., & Kuznetsov, N.A. (2023). Attentional focus effects on joint covariation in a reaching task. Human Movement Science, 89, Article 103089.
Kearney, P.E. (2015). A distal focus of attention leads to superior performance on a golf putting task. International Journal of Sport and Exercise Psychology, 13(4), 371–381.
Krajenbrink, H., van Abswoude, F., Vermeulen, S., van Cappellen, S., & Steenbergen, B. (2018). Motor learning and movement automatization in typically developing children: The role of instructions with an external or internal focus of attention. Human Movement Science, 60, 183–190.
Lamers James, R.G., & O’Connor, A.R. (2023). Impact of focus of attention on aiming performance in the first-person shooter videogame Aim Lab. PLoS One, 18(7), Article e0288937.
Lola, A., Tzetzis, G., Manou, V., & Aleksandropoulu, S. (2022). Attentional focus on learning fundamental movement skills in children. Physical Activity Review, 10(1), 60–67.
Ma, L., Marshall, P.J., & Wright, W.G. (2022). The impact of external and internal focus of attention on visual dependence and EEG alpha oscillations during postural control. Journal of NeuroEngineering and Rehabilitation, 19(1), Article 81.
MacMahon, C., Schücker, L., Hagemann, N., & Strauss, B. (2014). Cognitive fatigue effects on physical performance during running. Journal of Sport and Exercise Psychology, 36(4), 375–381.
Macnamara, B.N., & Burgoyne, A.P. (2022). Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. Psychological Bulletin, 149(3–4), 133–173.
Markwell, L.T., Makaruk, H., Frost, V., & Porter, J.M. (2023). Directing attention externally produces consistent vertical jump assessment results. International Journal of Exercise Science, 16(5), 448–457.
Markwell, L.T., Nolan, R., Brown, B., Makaruk, H., & Porter, J.M. (2022). Altering focus of attention effects isometric muscular endurance and heart rate during fitness testing. Measurement in Physical Education and Exercise Science, 26(3), 221–231.
Maxwell, J.P., & Masters, R.S.W. (2002). External versus internal focus instructions: Is the learner paying attention? International Journal of Applied Sports Sciences, 14(2). 70–88.
Mazza, A., Cimo, V., Valenzuela, K.A., & Will, W.U. (2022). The effect of neuropriming and focus of attention on amateur standing long jump performance. International Journal of Exercise Science, 15(1), 1472–1480.
McNicholas, K., & Comyns, T.M. (2020). Attentional focus and the effect on change-of-direction and acceleration performance. The Journal of Strength & Conditioning Research, 34(7), 1860–1866.
Mullen, R. (2007). Attentional focus and motor learning: Some caveats and cautions. E-Journal Bewegung und Training, 1, 39–40.
Neumann, D.L. (2019). A systematic review of attentional focus strategies in weightlifting. Frontiers in Sports and Active Living, 1, Article 7.
Neumann, D.L., Olive, A., Moffitt, R.L., & Piatkowski, T. (2022). Switching attentional focus across internal and external cues improves performance in a rowing task in novices. Psychology of Sport and Exercise, 61, Article 102195.
Nutpiboonmongkol, P., & Benjapalakorn, B. (2022). Effects of instructed attention focus on accuracy in golf pitch shot. Journal of Exercise Physiology Online, 25(3), 60–69.
Page, M.J., Moher, D., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C.D., Shamseer, L., Tetzlaff, J.M., Akl, E.A., Brennan, S.E., Chou, R., Glanville, J., Grimshaw, J.M., Hróbjartsson, A., Lalu, M.M., Li, T., Loder, E.W., Mayo-Wilson, E., McDonald, S., ... McKenzie, J.E. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. British Medical Journal, 372, Article n160.
Parr, J.V.V., Gallicchio, G., Canales-Johnson, A., Uiga, L., & Wood, G. (2023). Cortical, muscular, and kinetic activity underpinning attentional focus strategies during visuomotor control. Psychophysiology, 60(6), Article e14249.
Pompa, D., Carson, H.J., Beato, M., di Fronso, S., & Bertollo, M. (2024). Attentional focus effects on lower-limb muscular strength in athletes: A systematic review. The Journal of Strength & Conditioning Research, 38(2), 419–434.
Porter, J.M., Nolan, R.P., Ostrowski, E.J., & Wulf, G. (2010). Directing attention externally enhances agility performance: A qualitative and quantitative analysis of the efficacy of using verbal instructions to focus attention. Frontiers in Psychology, 1, Article 216.
Quartiroli, A. (2024). The Delphi technique in sport, exercise, and performance psychology: Extensive scoping review, insights, and recommendations for scholars. Sport, Exercise, and Performance Psychology. Advance online publication.
Ranganathan, R., Lee, M.H., & Krishnan, C. (2022). Ten guidelines for designing motor learning studies. Brazilian Journal of Motor Behavior, 16(2), 112–133.
Ranganathan, R., Tomlinson, A.D., Lokesh, R., Lin, T.-H., & Patel, P. (2021). A tale of too many tasks: Task fragmentation in motor learning and a call for model task paradigms. Experimental Brain Research, 239(1), 1–19.
Reynolds, W.M. (1982). Development of reliable and valid short forms of the Marlowe-Crowne Social Desirability Scale. Journal of Clinical Psychology, 38, 119–125.
Richer, N., Saunders, D., Polskaia, N., & Lajoie, Y. (2017). The effects of attentional focus and cognitive tasks on postural sway may be the result of automaticity. Gait & Posture, 54, 45–49.
Sakurada, T., Yoshida, M., & Nagai, K. (2022). Individual optimal attentional strategy in motor learning tasks characterized by steady-state somatosensory and visual evoked potentials. Frontiers in Human Neuroscience, 15, Article 784292.
Sawai, S., Fujikawa, S., Murata, S., Abiko, T., & Nakano, H. (2022). Dominance of attention focus and its electroencephalogram activity in standing postural control in healthy young adults. Brain Sciences, 12(5), Article 538.
Sherman, D.A., Lehmann, T., Baumeister, J., Gokeler, A., Donovan, L., & Norte, G.E. (2021). External focus of attention influences cortical activity associated with single limb balance performance. Physical Therapy, 101(12), Article pzab223.
Siltanen, S., & Bottas, R. (2022). Instructions for external focus of attention improved taekwondo kicking performance only among less skilled youth. Perceptual and Motor Skills, 129(3), 787–799.
Slovák, L., Sarvestan, J., Iwatsuki, T., Zahradník, D., Land, W.M., & Abdollahipour, R. (2023). External focus of attention enhances arm velocities during volleyball spike in young female players. Frontiers in Psychology, 13, Article 1041871.
Stoate, I., & Wulf, G. (2011). Does the attentional focus adopted by swimmers affect their performance? International Journal of Sports Science & Coaching, 6, 99–108.
Toner, J., & Moran, A. (2011). The effects of conscious processing on golf putting proficiency and kinematics. Journal of Sports Sciences, 29, 673–683.
Vargas-Molina, S., García Sillero, M., Salgado, U., Carbone, L., & Martín-Rivera, F. (2022). Effect of attentional focus on squats on lactate production and rating of perceived exertion in trained men. Retos: Nuevas Perspectivas de Educación Física, Deporte y Recreación, 47, 452–457.
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: A methodological systematic review. BMC Medical Research Methodology, 19, Article 132.
Woodard, K.F., Markwell, L.T., & Fairbrother, J.T. (2021). Effects of an expert-modeled attentional focus cue structure on skilled jump rope performance and learning. Human Movement Science, 80, Article 102889.
Wulf, G. (2013). Attentional focus and motor learning: A review of 15 years. International Review of Sport and Exercise Psychology, 6, 77–104.
Wulf, G., Höß, M., & Prinz, W. (1998). Instructions for motor learning: Differential effects of internal versus external focus of attention. Journal of Motor Behavior, 30(2), 169–179.
Wulf, G., Lauterbach, B., & Toole, T. (1999). The learning advantages of an external focus of attention in golf. Research Quarterly for Exercise and Sport, 70(2), 120–126.
Wulf, G., McNevin, N.H., Fuchs, T., Ritter, F., & Toole, T. (2000). Attentional focus in complex skill learning. Research Quarterly for Exercise and Sport, 71, 229–239.
Yamada, M., Higgins, L.Q., & Raisbeck, L.D. (2021). Attentional focus and individuals’ perceptions: A systematic review. Journal of Motor Learning and Development, 9(2), 327–343.
Yamada, M., Lohse, K.R., Rhea, C.K., Schmitz, R.J., & Raisbeck, L.D. (2022). Practice—Not task difficulty—Mediated the focus of attention effect on a speed–accuracy tradeoff task. Perceptual and Motor Skills, 129(5), 1504–1524.
Yamada, M., & Raisbeck, L.D. (2021). The autonomy and focus of attention strategies under distraction: Frequency and sample entropy analyses in a dynamic balance task. Human Movement Science, 80, Article 102882.
Yamada, M., Raisbeck, L.D., & Porter, J.M. (2021). The effects of using imagery to elicit an external focus of attention. Research Quarterly for Exercise and Sport, 92(3), 559–565.
Yeh, T.T., Cinelli, M.E., Lyons, J., & Lee, T.D. (2016). Using a manipulation check to uncover age-related difference in focus of attention instruction during a balance task. Experimental Aging Research, 42(3), 307–313.
Zhuravleva, T.A., & Aiken, C.A. (2023). Adopting a holistic focus of attention promotes adherence and improves performance in college track and field athletes. Human Movement Science, 88, Article 103055.
Zhuravleva, T.A., Aiken, C.A., Becker, K.A., Lin, P.-C., & Sampson, J.J. (2023). The use of a holistic focus of attention to improve standing long jump performance among NCAA track and field athletes. International Journal of Sports Science & Coaching, 18(4), 1074–1081.
