A common discussion point among experienced sports scientists, especially those who serve on journal editorial boards with opportunities to vet incoming manuscripts, is that studies of supplements and sports performance seem to be considered an entry point to the world of sports nutrition research. To honors students with limited time and money for project work, or to a new laboratory looking to get some “runs on the board” with a similar lack of resources, it might seem that an ergometer, a stop watch, and a bottle of magic pills provide the means to become a researcher. Unfortunately, the lack of understanding of the complexity of supplement research, and the lack of academic recognition for doing it well, often means that the quality of research is compromised. Notwithstanding, this process may also be abused by unscrupulous manufacturers who want to build an aura of credibility for their products or by predatory journals who publish all manuscripts as a commercial business model rather than an appropriate outcome of peer review. Indeed, it is likely that this area of sports nutrition suffers more than others from poor methodologies, a low bar for publication, and publication bias. This review will address key issues in undertaking research on performance supplements, develop some guidelines for improving the methodologies of such work, and promote a better understanding of the limitations inherent within the current literature. The commentary will be limited to issues around sports foods and performance supplements, and their potential to directly enhance the training capacity and competition outcomes of serious athletes.
Although there are concerns around the indiscriminate use of supplements and sports foods by athletes, many bodies now extend a pragmatic acceptance to products which have passed a risk:benefit analysis of being safe, effective, and legal (i.e., not banned for use in sport), while also being appropriate to the athlete’s age and maturation in their sport (Burke & Cato, 2015; Maughan et al., 2018; Thomas et al., 2016). The success of this analysis is dependent on the quality of the evidence base on the supplement in question. Unfortunately, there are major challenges in making specific decisions around supplement use due to the frailties of the available literature and the enormous absence of systematic investigation of important topics (summarized in Table 1). This review will summarize these challenges and how to address them, albeit with the brevity needed to allow coverage of a wide number of issues.
Challenges in Making Evidence-Based Decisions Regarding the Efficacy of Performance Supplements and Sports Foods From the Available Research
| Issue | Comment |
|---|---|
| The vast number of supplements are untested. | There are millions of products—individual ingredients and/or combinations of ingredients in commercial products—that dwarf the opportunities for sports scientists to test them. |
| Few studies interrogate the specific and personalized scenarios in which athletes wish to apply supplement use. | Although there may be a popular interest in asking if Supplement X “works”, in reality, athletes wish to apply Supplement X to a specific race/competition or training program, with a unique set of challenges. Generic solutions may not always apply to specific scenarios. |
| Many available studies suffer from limitations in methodology which may mask a true interpretation of the results. | Research that is poorly conducted or interpreted may result in type 1 (incorrect attribution of a performance change) or type II (failure to detect a true performance change) errors. |
| Few studies investigate the simultaneous use of a number of supplements to test the interaction of products. | Athletes often use a number of individual supplements at the same time, or commercial products that contain multiple ingredients. The opportunity for ingredients to be additive, neutral, or counteractive when used in combination is rarely investigated. |
| Few studies investigate the repeated use of supplements for successive performance bouts. | Although in many sports, the winner of an event is decided by a series of races/stages/matches, the efficacy of a supplement protocol when it is repeated within a day(s) is rarely investigated. |
| Few studies investigate the effect of individual responses to supplement use. | The effect of known characteristics (e.g., age, sex, caliber of performance, training status) on the efficacy of most supplements has rarely been systematically studied. Individual variability due to genetic or other differences is misunderstood and poorly represented in appropriate research. |
| Few studies can detect, and many incorrectly interpret, the effect of performance changes that would be meaningful to real-life sporting competition. | Few studies are conducted with methodologies or statistical power to detect meaningful changes in performance. However, many scientists and athletes/coaches misunderstand the concept of intra- and interindividual variability in performance and may over-interpret small changes/differences in performance in study results against the tiny margins that separate place-getters in real-life sporting events. |
Philosophies Underpinning the Proof of Performance Benefits
Substantiating the claims made about performance supplements and sports foods is a difficult process. To various audiences, “proof” comes in different forms. Figure 1 provides a hierarchical model of the suggested strength of the evidence provided by different information sources. The most common types of information around supplement efficacy come from models with the lowest rigor: anecdotes/observations from athletes and scientific/mechanistic hypotheses which explain how a supplement might target a critical/limiting factor in performance. Despite some hesitancy about the scientific credibility of this type of information, there have been situations in which science has later come to support supplement practices observed among athletes. Such practices were earlier dismissed as being incorrect or unlikely to achieve performance benefits—in other words, scientists were only able to confirm and explain outcomes which athletes had already worked out for themselves through trial and error (Cox et al., 2002). However, caution is also advised, since individual experience and correct attribution can be a slow and inaccurate process. Furthermore, the supplement world is filled with testimonials from athletes and scientists underpinned by incentives that render them less than objective; this is not always transparent.
—Different types of evidence that might be applied to decisions made about the efficacy of use of performance supplements.
Citation: International Journal of Sport Nutrition and Exercise Metabolism 28, 2; 10.1123/ijsnem.2017-0325
Systematic reviews and analyses, which synthesize the outputs of many studies into a pooled estimate of outcome are at the top of the evidence hierarchy. Indeed, such information is available on products such as caffeine (Burke, 2008; Conger et al., 2011; Glaister & Gissaine, 2017), including its specific delivery in energy drinks (Souza et al., 2017), bicarbonate (Carr et al., 2011b), creatine (Branch, 2003; Mujika & Padilla, 1997), b-alanine (Hobson et al., 2012; Saunders et al., 2016), nitrate/beetroot juice (Hoon et al., 2013; McMahon et al., 2017), antioxidant supplements (Braakhuis & Hopkins, 2015), N-acetylcysteine (Rhodes & Braakhuis, 2017), and polyphenols (Somerville et al., 2017). Other meta-analyses target specific uses of sports products, such as, in the case of sports drinks and gels, carbohydrate intake during endurance sport (Stellingwerff & Cox, 2014). Although these reviews address the challenge of small sample sizes in sports science projects, and often contribute to a scientific and pragmatic acceptance of supplement practices in sport, there are caveats to their use in producing both generalizable and specific proof of efficacy. The recommendations around conducting and interpreting meta-analyses are summarized by Ioannidis and Lau (1999) and include the important observation that a meta-analysis cannot improve on the quality or reporting of the original studies; in other words, the end product is only as good as the materials that went in to building it.
The underpinning of a meta-analysis is the controlled scientific trial, which may vary subtly according to the nature of the starting question (i.e., whether the sports scientist begins with an interest in a specific supplement and wants to identify a sporting event in which it might be useful or whether the sports scientist is targeting a specific sporting event and wants to investigate whether a supplement might enhance performance). Indeed, although the consumer may think that the quest is to seek answers to a generalizable question (Does supplement X work?), the real topic of interest to many athletes is more specific, and, in many important cases, very specific (see Figure 1). In the latter scenario, the complexity of the intended application of the supplement overwhelms the opportunity to undertake a scientific trial, both on methodological and economic grounds. Nevertheless, we note that some recent supplement studies have targeted the challenges of real-life performance by either conducting the study with high-caliber athletes in a simulated competition in field conditions (Del Coso et al., 2013b, 2014, 2016; Pérez-López et al., 2015), actual competition (Del Coso et al., 2013a; Portillo et al., 2017), or as a laboratory simulation of a real-life event (Lane et al., 2014) incorporating real-life practices and environments. Such studies may address some of the real-world questions around the use of performance supplements, but when the unique scenarios of specific athletes (individual responses) or events (specific environment, timetable, workload characteristics, potential use of several supplements) are outside the scope of the available literature, evidence of performance benefits may need to come from individual or small group case studies. In such instances, repeated trials with the supplement under event conditions—even using a placebo treatment in some trials—may help the athlete to develop confidence that their supplement protocol is useful. Recommended methodologies for single/small group case studies include repeated baseline performances before the introduction of the supplement, or alternating a series of presentation and absence of the supplement (McDonald et al., 2017).
The Scientific Trial
The controlled scientific trial is considered the ‘gold standard’ for investigating the effects of performance supplements and sports foods on sports performance. In medical/clinical situations, the ideal version is the randomized controlled trial (RCT), in which a large number of subjects are randomly allocated to receive an experimental or placebo treatment under standardized conditions. In sports science, however, practical issues may require some variations to the ideal design (see Table 2). In particular, the scarcity of elite athletes (by definition), limited resources, and conditions around subject availability often means that sample sizes for supplement studies are small and/or drawn from subelite performers. While these athletes make up the majority of the market for performance supplements, there is a need for more investigation of truly world-class athletes for whom the benefits of enhanced performance may be more important, and likely harder to achieve.
Elements in Methodology of Controlled Studies of Athletes Involving Performance Supplements and Sports Foods
| Element of Methodology | General Comments | Specific Comments Around Supplement Studies |
|---|---|---|
| Design | ||
| Crossover design: Compares outcomes in a single group who receive both supplement and placebo, with different order of treatment being balanced within group. Can be fully controlled (baseline and treatment measurement for both experimental and placebo) or simple crossover (no baseline measurement) when time or resources are limited. | Strong statistical design: Each athlete acts as their own control, reducing intersubject variability and interference from co-variates. Requires fewer athletes than parallel group design, crossover takes longer to complete. May require a washout period to avoid carryover effects of supplement on next test. Can enhance recruitment: athletes know all will eventually trial the supplement. | Useful when target group has limited numbers (e.g., working with a small team or focusing on elite athletes). Suited to studies where multiple treatments are desired (e.g., different doses of a supplement, interaction of several supplements). Generally less suitable when supplements require lengthy (weeks) washout (e.g., creatine, B-alanine) or athlete availability or training status changes. |
| Parallel group design: Compares outcomes in a group who receives the supplement and a group who receives a placebo. Pre- and posttreatment in both groups. | Has less power than crossover design due to greater subject variability. Requires more subjects than crossover design. Typically takes a shorter time to complete. May reduce recruitment response: Athletes know that some will not receive the supplement. | Requires careful allocation of athletes into groups according to key characteristics. Suited to studies of chronic supplement use or supplements requiring lengthy washout periods. Useful for opportunities to work with a large team for a short time period. |
| Parallel group design allocation models (True random allocation is typically not used with small samples: Important variables are unlikely to be evenly distributed across groups.) | ||
| Stratified allocation: Categorizes athletes according to predetermined characteristics (e.g., sex, caliber, training status). Randomly allocates within subgroups to treatment/placebo before first trial. | Allocates into treatment groups based on known characteristics prior to the study. If sufficient statistical power, may allow covariate analysis to determine magnitude of effect arising from these characteristics. | Pragmatic for studies with longer time frame and continuous enrolment of athletes. Nonstratified characteristics may not be equally spread between groups leading to pretrial differences; some may be considered important (e.g., pretest performance). |
| Pair-matched or matched group allocation: Baseline or characterization testing ranks or matches athletes into pairs according to important characteristics, then assigns to treatment or placebo group. | May require all pretesting to occur before study treatment can be commenced. May be difficult to identify a single characteristic to match against; in some cases, an amalgamation of several characteristics may be possible. | Valuable when important characteristics (e.g., aerobic capacity, pretest performance) vary between athletes: Distributing differences across groups harmonizes the baseline values. Practical for studies conducted as a discrete project over a single timeframe. |
| Presentation of treatment | ||
| Double-blind placebo control: Neither the athlete nor researcher know the treatment allocation until the end of the study. | Provides inert product (placebo) with identical presentation to the treatment to some or all athletes. Controls for the benefits associated with the placebo effect or observer bias. | Difficulties arise with finding placebo for supplements with distinct characteristics (e.g., the taste of ketone ester) or distinct effects (e.g., B-alanine and tingling, caffeine and increased heart rate, creatine loading and weight gain). Poststudy information regarding athlete beliefs around the treatment received can check the success of blinding. If treatment cannot be blinded, observer bias during testing should be reduced by keeping researchers unaware of treatment allocation (single-blinded protocol). |
The goal of any experimental study is to investigate the effects of the independent variable (e.g., the performance supplement or sports food) on a range of dependent or outcome variables (e.g., performance and perhaps some mechanistic parameters), while standardizing or controlling for factors that might otherwise interfere with the outcomes (confounding variables), and potentially probing whether other factors systematically alter them (covariates). Even when the research goal is to investigate the application of a supplement for sports performance, it can be valuable to focus on measuring mechanistic issues, since these can explain or validate observations of performance change. Furthermore, understanding the metabolic or mechanistic events that underpin the performance change can enhance our ability to apply the supplement use beyond the scenarios that have been directly investigated. For example, it may allow the supplementation protocol to be manipulated for an improved outcome or to suit the specific needs of another scenario of use (i.e., changing the dose or timing of intake based on information about the time-course of metabolic changes). It is beyond the scope of this review to fully explore the characteristics of good research design, however, some commentary on designing and conducting studies to optimize the involvement of these variables is provided in Table 3, as well as other specific reviews on dietary standardization prior to performance trials (Jeacocke & Burke, 2010), external sources of motivation (Halperin et al., 2015), and the organization of reliable and valid measurements of performance (Currell & Jeukendrup, 2008). The review will finish by considering some of the themes that challenge research and its interpretation identified in Table 1.
Discussion of Factors Involved in Studies of Supplement and Performance
| Type of Factor | Examples of Factor | Specific Guidelines for Supplement Studies |
|---|---|---|
| Supplement protocol (independent variable) | ||
| Supplement should be investigated using a robust protocol | Acutely applied supplements | Conduct independent testing of commercial products to ensure that ingredients of interest are present in the stated amount, without “contamination” from other ingredients, including banned substances. Choose supplement protocols that are known or have a high chance of achieving desired metabolic/mechanistic changes that might enhance performance. If possible, monitor characteristics that confirm the changes |
| Chronically applied supplements | As above, plus measure compliance to the supplementation protocol. | |
| Confounding variables | ||
| Should be controlled or standardized to improve the reliability of the supplement effect or the performance measure. Note that some studies may be sufficiently powered to allow these factors to be investigated as a covariate: After the study is completed, statistical analysis may be able to determine if differences in factors that occurred during the study explain or alter the effect of the supplement. Alternatively, can be included in the study as an independent variable: The factor may be manipulated (e.g., temperature, training status, or nutritional status) to investigate if there is a consistent and meaningful effect on the study outcome. | Chronic nutrition status: Differences in general nutritional status between subjects or changes between trials may alter response to some supplements or performance outcomes. | Avoid or correct low energy availability, poor iron status, or other nutrient deficiencies as these may alter performance of some protocols and/or the response to supplements. Note that vegetarians may respond differently to supplemental ingredients that are low in their diets (e.g., creatine, carnitine, and b-alanine). Ensure athletes are not using other chronically consumed performance supplements (e.g., creatine, B-alanine)—: where this is not possible (e.g., real-life trials), standardize and note their intake. |
| Acute nutrition status: Differences between subjects or changes between trials may alter response to some supplements or performance outcomes. Simultaneous use of other performance supplements may alter the effects of the supplement of interest or performance outcome. If results are to be applied to real-life sport, nutrition practices around performance protocol should mimic these. | Control/standardize carbohydrate (CHO) intake in the day(s) prior to trials in which performance is dependent on CHO availability. Where performance protocols are dependent on hydration status, control fluid intake on the evening before, and hours before a trial. Restrict intake of alcohol during 24 hr pretrial. Standardize rather than withdraw caffeine use in the days prior to a trial. Restrict intake of caffeine and other performance supplements on the day of trial, unless effect of standardized co-ingestion or real-life practices is desired. Standardize pretrial meal and CHO/ fluid intake during exercise protocol to mimic real-life practices or sports nutrition guidelines. | |
| Training status: Differences in overall training status or immediate levels of fatigue may affect response to some supplements or performance outcomes. | Choose athletes who represent the population to whom the study results are targeted. Conduct studies when athletes are ready to perform reliably and be aware of changes in training status during chronic studies. Standardize training in the day(s) before performance trials to standardize readiness to perform well/reliably. | |
| Time of day: Diurnal changes in metabolic and hormonal parameters may influence the effect of the supplement or the effect on performance. | Standardize the time of day when the study is performed—at the least, schedule each athlete to undertake repeated trials at similar times. If the study is focused on a specific event, consider undertaking trials at the time of day that it will be performed. | |
| Environmental conditions: Differences in temperature, humidity, airflow, altitude may affect performance outcomes. | Standardize and report laboratory conditions. Keep conditions “neutral” unless the study aim is to investigate effect of different environments on supplement outcomes. Where it is not possible to standardize field conditions, note these and treat them as a covariate. | |
| “External support” environment: External factors or cues may alter motivation or performance. | Remove or standardize external factors that may alter performance (e.g., music, encouragement, awareness of time). If using incentives to promote performance (e.g., prize money), ensure that these are applied equally between trials and subjects. | |
| Mechanistic variables | ||
| Measurement of mechanistic variables can strengthen the interpretation of performance results by explaining the “cause” of performance changes. However, it may interfere with performance, directly (physical or mental interruption) or indirectly (providing cues about elapsing of time). | Unfortunately, the most informative data (blood metabolites, muscle metabolites, substrate use from pulmonary gases) are the most “invasive” to collect. Separate studies might focus on these variables; nevertheless, performance-focused studies may still collect some mechanistic data if it is well chosen and well timed. | Balance the pros and cons of collecting “invasive” samples during a performance protocol—limit collection to before and after the protocol, or to a standardized preload or within-protocol monitoring period. Limit the impact of monitoring on performance—keep it brief or integrate it into a protocol mimicking real-life competition (e.g., the warm up, or race phases when the athlete might be in approximate steady state before mounting a performance burst). Collect data according to a percentage of task completion (e.g., every 25% of distance) rather than time splits (e.g., every 20 min) to avoid giving clues to the athlete about their performance. |
| Performance variables | ||
| The performance measure should be valid and reliable in order to detect meaningful changes due to the performance supplement. | Laboratory protocols: Reliability and validity of performance protocols determine the ability to detect small but important changes in performance that are of relevance to real-life competition. | Control confounding variables as much as possible. Recruit higher-caliber athletes with extensive experience of pacing strategies and greater performance reliability. Schedule familiarization trial(s) of the performance protocol—even experienced athletes will require familiarization with an unpracticed laboratory task. Know the reliability of the performance task to assist in interpreting the results. Use a performance protocol that mimics real-life sport as far as possible, including tasks that are part of an athlete’s normal performance monitoring. Further enhance validity by using trial conditions that mimic real life or recommended strategies. |
| Field studies: Field studies and, in particular, real-life competition trials maximize validity of performance, but often sacrifice loss of experimental control and reliability. | Control confounding variables as much as possible. If a crossover design is used over several real-life competitions, ensure that competitors are split between supplement and placebo trials to account for different conditions. Be prepared to use a more sophisticated statistical analysis to account for differences in conditions. |
Simultaneous Use of Several Supplements
Athletes often report the simultaneous use of a number of different performance supplements (Erdman et al., 2007; Shaw et al., 2016). Some of these practices are not recommended; for example, indiscriminate polypharmacy or the use of multi-ingredient supplements with undeclared “proprietary blends” place the athlete at risk of unknown and potentially hazardous intakes of ingredients, including stimulants and banned substances. However, there is a legitimate interest in investigating the deliberate combination of several evidence-based supplements with the goal of optimizing performance benefits via an additive effect or positive interaction. Scenarios in which this might occur include the simultaneous use of several products with individual benefits in the acute scenario of competition, or the acute use of supplements in conjunction with a chronically applied supplement used to support training outcomes. A recent review of the available literature on such supplement combinations noted that even when the focus is restricted to a few performance supplements (e.g., caffeine, creatine, beta-alanine, bicarbonate, nitrate/beetroot juice, and phosphate), and the specific scenarios in which there is evidence-based justification of their use, a large number of permutations of supplement use combinations can be identified (Burke, 2017). For example, each of these six performance supplements, when used in isolation, could theoretically provide a benefit to the preparation (beta-alanine, creatine) or performance (caffeine, beta-alanine bicarbonate, nitrate, phosphate) of a 4,000-m track cycling pursuit, 400-m swim, or 2,000-m rowing race (Burke, 2017).
Not surprisingly, the scientific literature, which has only just started to address supplement combinations, fails to provide evidence for an optimal protocol for combining the use of some or all of these supplements. Indeed, it would be almost impossible to conduct a study in which the independent and interactive effects of each of the combinations of these products could be tracked. As such, current research endeavors have typically limited the focus to the single and/or combined use of two products. Yet, such research is important, in light of the potential for a number of different types of interactions to occur. For performance supplements with related mechanisms, such as bicarbonate and beta-alanine supplementation, which target the fatigue associated with excessive muscle acidosis, combined use might be additive or neutral (Burke, 2017). Meanwhile, for those with independent mechanisms, combined use could be additive, neutral, or counterproductive. Indeed, the available studies report examples of each scenario, with interactive effects including an additive effect of bicarbonate and beta-alanine supplementation on performance of a combat sports simulation (Tobias et al., 2013), bicarbonate and caffeine supplementation combining to increase the performance benefits of another judo simulation (Felippe et al., 2016), but bicarbonate counteracting the benefits of caffeine on rowing ergometer performance due to gut disturbances (Carr et al., 2011a). A Latin squares application of treatments in a crossover-designed study can provide a robust and practical framework for investing the single and combined effect of two acutely administered products, with priority given to undertaking such studies on supplement combinations which hold the most potential for interaction effects. Meanwhile, the integration of larger numbers of products into the athlete’s program might be practically managed using single case and small group observations, with intuition and experience also guiding the process (see Figure 1).
Repeated Use of Supplements
In many sports, competition outcomes are decided through a series of heats and finals, stages in a race, or games in a tournament. In other sports, gifted athletes may compete in several events during the competition program (e.g., Michael Phelps’ haul of eight gold medals at the 2008 Beijing Olympic Games required 17 races over an 8-day program, while Usain Bolt raced seven times in 7 days, including twice in the same session at the 2016 Rio Olympic Games). In some cases, the interval between competitive bouts is measured in hours and may fall within the half-life of a supplement or the body’s return to normal physiological status or homeostasis following the event. Whether the evidence-based use of a supplement is similarly efficacious when immediately repeated for a subsequent event is, therefore, of practical importance to competitive sport. However, in these instances at least three different issues might need to be considered: (1) subsequent use of the supplement might require a different protocol to restore the physiological advantage or to meet the logistical requirements of competition spacing; (2) desensitization of physiological systems may render the subsequent use of a supplement less effective; (3) residual fatigue left from enhanced performance in the first event may carry over to the subsequent event (Burke, 2017). The corollary to these possibilities include an altered protocol (timing and dose), the decision not to use the supplement on all occasions, or the need to modify pacing strategies for earlier events.
The sparse literature on the repeated use of well-supported performance supplements includes evidence that a period of abstinence from caffeine use does not change the magnitude of its benefits to a subsequent test of sports performance (Irwin et al., 2011). While the practical consequences of this study are that athletes do not need to include a period of withdrawal before using caffeine to enhance competition performance, they also suggest that caffeine will be effective if used on repeated occasions. Indeed, caffeine supplementation was shown to be equally effective, when repeated, 24 hr apart, to enhance the performance of two cross-country ski time-trials (Stadheim et al., 2014). This benefit occurred despite increased muscle damage and soreness from the first bout, attributed to the greater exercise effort made possible by caffeine use; presumably, the repeated use of caffeine was able to mask these symptoms during the second competitive bout.
Repeated bicarbonate supplementation, using a chronic daily loading protocol, has been studied in a protocol simulating a swimming carnival with successive day races (Joyce et al., 2012), and a rowing regatta where races were separated by 48 hr (Carr et al., 2012). In these studies, the bicarbonate loading protocol failed to provide the expected performance benefits in any of the exercise bouts, suggesting problems with the study design. By contrast, the repetition of acute bicarbonate loading for five consecutive days maintained benefits to high-intensity cycling capacity (Mueller et al., 2013), suggesting that further investigation is merited. Beetroot juice is the only supplement that has been studied in a design simulating a within-session repeated use; this investigation targeted the track cycling team pursuit, where semifinal and final races on the Olympic Games can be separated by a 75-min interval (Hoon et al., 2014). Unfortunately, different attempts to manipulate the typical beetroot juice protocol (intake 2–2.5 hr preevent) to suit this timetable were marred by use of a supplement dose that would now be considered too low to be effective, particularly in high-caliber athletes. Clearly, further work is required around the repeated use of supplements to cover the various practical and metabolic issues encountered during competitive schedules where cumulative attempts to augment performance through nutritional supplementation are possible.
How Should Study Results Be Interpreted?
The presentation and interpretation of data from studies involving small sample sizes requires special care; the reader is referred to several recent reviews which cover a range of valuable insights that are beyond the scope of the current paper (Drummond & Tom, 2011; Weissgerber et al., 2015). Here we will simply focus on several recent trends in the interpretation of studies of performance supplements that are relevant to the current discussion—the provision of individual performance outcomes within group data and commentary around the significance of these results in terms of typical outcomes of real-life sporting competitions. Although there is merit behind these concepts, the execution of such analyses is frequently marred by the failure of researchers (and, presumably, readers) to distinguish normal day-to-day variability in performance from “individual responses” to a supplement, or to understand that the small differences between competitors in real-life sport do not determine the size of the benefit a supplement needs to achieve to be considered effective. Figure 2 provides an example of such data presentation.
—Results of a mythical study of the effects of Supplement X on 100-m butterfly performance, as they are often presented and misinterpreted by sports scientists.
Citation: International Journal of Sport Nutrition and Exercise Metabolism 28, 2; 10.1123/ijsnem.2017-0325
Are We Testing the Athlete’s Definition of Improvement?
To an athlete or coach, a performance improvement measuring fractions of seconds or meters might seem meaningful, since it is often the margin between the gold and silver medal winners, or the difference between a podium finish and failing to make a final. This helps to explain why supplements that promise a performance boost seem alluring, even if they only achieve a fraction of their claims. By contrast, traditional sports science research uses a framework to detect changes/differences in performance based on their statistical likelihood of occurrence, with a typical rate of acceptance of nonexistent effects of 5%, and failed detection of a real effect of 20%. However, when this approach is coupled with a small sample size and a performance protocol with low reliability, the statistical power will detect only a sizeable change/difference in performance, with anything less being traditionally dismissed as “not significant”.
Hopkins and colleagues have popularized a different approach, known as magnitude-based inferences, which represents a middle ground between these extreme views of what is meaningful for sports performance (Hopkins et al., 1999). The focal point of this approach is an understanding of the importance of an athlete’s day-to-day variability in performance. This variability (or coefficient of variation [CV]) in performance is typically larger than the difference between competitors in an event, and predicts that a different outcome would occur if a sporting competition was re-run without any intervention. According to modeling, however, an improvement equal to ∼0.3 of the CV of performance variability for that event could produce a meaningful difference to the event outcome (Hopkins et al., 1999). This does not guarantee that the recipient will move from being a finalist to a certain winner, but, rather, it may simply increase their likelihood of winning by an absolute increase of ∼10% (for example, taking the athlete who has a 40% chance of winning a race to new odds of a 50% chance) (Hopkins et al., 1999). Sports scientists are encouraged to report their outcomes as a percentage change in the measure of athletic performance, use 90% to 95% confidence limits to describe the likely range of the true effect of the treatment in this scenario, and to consider the smallest worthwhile change (i.e., 0.3 CV performance) for the protocol/event (Batterham & Hopkins, 2006). While this approach has been criticized by traditional statisticians (Welsh & Knight, 2015), many applied sports scientists feel comfortable with the ability of magnitude-based inference techniques to provide qualitative determinations of the likely true change in performance with real-world significance (Buchheit, 2016). Furthermore, a recent re-analysis of data from a typical study of sports performance (e.g., small sample size with small changes in dependent variables), using a fully Bayesian statistical approach, found that the outcomes were consistent with those obtained using the previously-published magnitude-based inferences approach (Mengersen et al., 2016). This suggests that a number of options may be suitable when exploring the potential influence of supplement interventions on performance outcomes.
Analysis of the reliability of performances of top athletes across a large number of sports (Malcata & Hopkins, 2014) has shown that this varies according to the features of their sport and the level of the competitor. Skeleton and 1,000-m speed-skating times were shown to have the lowest within-athlete variability (CV of 0.15% and 0.4%, respectively), apparently because of the effect of the initial phase of the race on race dynamics. Times in sprint and endurance sports also have relatively low variability (0.6–1.4%), reflecting the important contribution of mean power output to performance. Meanwhile there is higher variability in sports performed against water or wind resistance due to uneven effects on individual athletes. Sports requiring explosive power in a single effort, such as field events and weightlifting, have larger CVs for their performance measures (1.4–3.3%), likely reflecting substantial contributions of skill. Sports with the greatest within-athlete variability (∼50%) were those with subjective scores (e.g., surfing). Overall, elite athletes are more reliable than their less-successful or less-well-trained counterparts. Sports scientists who are interested in the effects of a supplement on a particular event and population should try to determine the specific CV of their group to delineate the size of a worthwhile intervention. Overall, however, an intervention that can enhance performance by 0.5–1% is likely to have real-life significance in determining the outcome of competition, while improvements of 2–3% are generally robust in their effect (Malcata & Hopkins, 2014).
Do Individual Differences in Response to Supplements Exist?
Notwithstanding daily performance variability, the real-life experiences of athletes and scientists include encounters with individuals who respond differently to some performance supplements than their peers. This calls for measures in research and field settings to confirm these observations, as well as to explain their basis. Some differences in responsiveness to a supplement arise from differences in background nutritional status or the body’s own supplement stores; for example, creatine loading protocols have greater impact in vegetarians, who have lower muscle stores due to their lower dietary creatine intake (Watt et al., 2004). Furthermore, caffeine supplementation produces a smaller benefit in endurance protocols in which carbohydrate availability is high, since there is less need to mask fatigue (Conger et al., 2011). Other characteristics that may alter responsiveness to some supplements include sex and training status/athletic caliber.
Males and females differ in many ways that are important for sports nutrition outcomes—for example, body size, body composition, and hormonal profiles that affect metabolism. Just as macronutrient targets are now often given relative to body mass or fat free mass, in some cases, doses of supplements are also scaled to body size (e.g., caffeine dose of 3 mg/kg, bicarbonate dose of 300 mg/kg). Whether the hormonal characteristics of the female athlete and their fluctuations over the menstrual cycle, known to have measureable effects on physiology, create differences in the responsiveness of females to supplement strategies is of interest. In most cases, research on sport-specific uses of performance supplements concerns male subjects, and direct comparisons of male and female responses to supplement use is rare. Despite gaps in the evidence base regarding sex differences with respect to supplement use, we generally believe that female athletes respond similarly to their male counterparts when circumstances are matched. Regardless, it is clear that further scrutiny is deserved.
The concept that elite and highly-trained athletes might respond differently to sports nutrition strategies than their lesser-trained or less-successful counterparts has been recently examined in relation to beetroot juice/nitrate supplements (Hultström et al., 2015; Jonvik et al., 2015). As usual, it can be dangerous to make firm conclusions from a literature base that is limited in size and quality. Nevertheless, there is evidence to support the hypothesis from single studies in which the benefits of supplementation were inversely correlated with maximal aerobic capacity (Porcelli et al., 2015), as well as from a meta-analysis of studies which showed greater effect sizes of performance enhancement with lower-caliber athletes (Hoon et al., 2013). Furthermore, there are credible explanations of how highly-competitive sports people might differ from the rest of the athletic population due to the effects of years of training adaptations or the genetic differences that have selected them to be so suited to their event. For example, in the case of nitrate supplementation, a potential reduction in the benefits of enhancing the activity of the nitrate-nitrite-NO pathway in elite athletes might be explained by different muscle fiber type composition, greater muscle capillarization, and adaptation, which reduce the development of hypoxia and metabolic acidosis, and a more developed pathway to produce nitric oxide from arginine (Jones, 2014). However, since studies of high-caliber athletes (Peeling et al., 2015) have reported performance enhancement with nitrate supplementation, albeit in some specific scenarios, this suggests that even if nitrate supplementation is more valuable to lesser-trained or lower-caliber athletes, there are perhaps certain sports, events, and conditions in which it may also assist elite competitors.
Genetic differences are also likely to explain some of the observed differences in supplement responses between individual athletes. For example, the variable type, number, and site of adenosine receptors around the body can explain responsiveness to the adenosine antagonist caffeine, while differences in liver metabolism can explain differences in caffeine’s half-life (Yang et al., 2010). Indeed, there is already preliminary evidence in well-trained cyclists that different versions of a single nucleotide of a gene associated with cytochrome P450 (CYP1A2), a family of proteins involved in hepatic metabolism of drugs, are associated with “high responder” or “minimal responder” outcomes when caffeine is taken to enhance performance of a cycling time-trial (Womack et al., 2012, 2015). However, as recently reviewed (Pickering & Kily, 2017), responsiveness to caffeine is a complicated issue, likely underpinned by a combination of genetic, environmental, and epigenetic factors. Undoubtedly, genetic polymorphisms and their effect on supplement metabolism and activity are likely to be a fertile area of future research and may be incorporated into guidelines for personalized nutrition strategies. Of course, it should be remembered that many of the factors influencing responsiveness to caffeine can be manipulated (e.g., timing and amount of the dose). Therefore, it has been pointed out that participants who failed to show performance enhancement to a particular caffeine supplementation protocol should be considered as “failed to respond” to the specific scenario rather than “nonresponders” (Pickering & Kily, 2017).
Studies employing simple group analysis and small sample sizes are not appropriate for situations in which there is true variability in the size and direction of the response to an intervention. Ideally, studies employing large sample sizes, good control over confounding variables, and covariate analysis could be used to probe for individual responses; this approach will allow real changes to be detected and may also identify the characteristics of individuals that predict “response” and “absence of response”. It is useful to have metabolic or other mechanistic data to correlate changes in these features with changes in performance (Casey & Greenhaff, 2000). Future opportunities may also allow scientists to probe for genetic markers associated with differences in metabolism or response to a substance. In the meantime, undertaking repeated trials in the same individual to confirm the robustness of a measured response to a supplement is also valuable (Graham & Spriet, 1991; Talanian & Spriet, 2016) and may provide confidence regarding the presence of an “outlier” outcome. For the moment, few researchers appear to have either the resources or the belief in this concept to implement it in their studies. However, in individuals, it may be managed via the repeated case study approach.
Conclusion
Sports nutrition research should address the important questions that athletes and coaches need to consider in making evidence-based decisions about if and how to use a specific performance supplement. Unfortunately, the opportunity to undertake high-quality investigations is dwarfed by the number of products on the market and the complexities of the real-life scenarios of use that need to be explored. All such research should be underpinned by the use of performance tasks with high levels of reliability and validity. Issues that have been poorly addressed to date include the use of several supplements in combination and the use of the same supplement over successive events. Strategies to isolate the variability of benefits to individuals are also a topic for future interrogation. Nevertheless, models to conduct and interpret studies of performance supplements are provided and may help to address some of the current challenges and missing information.
References
Batterham, A.M., & Hopkins, W.G. (2006). Making meaningful inferences about magnitudes. International Journal of Sports Physiology and Performance, 1, 50–57. PubMed doi:10.1123/ijspp.1.1.50
Braakhuis, A.J., & Hopkins, W.G. (2015). Impact of dietary antioxidants on sport performance: A review. Sports Medicine, 45, 939–955. PubMed doi:10.1007/s40279-015-0323-x
Branch, J.D. (2003). Effect of creatine supplementation on body composition and performance: A meta-analysis. International Journal of Sport Nutrition and Exercise Metabolism, 13, 198–226. PubMed doi:10.1123/ijsnem.13.2.198
Buchheit, M. (2016). The numbers will love you back in return - I promise. International Journal of Sports Physiology and Performance, 11, 551–554. PubMed doi:10.1123/ijspp.2016-0214
Burke, L.M. (2008). Caffeine and sports performance. Applied Physiology Nutrition and Metabolism, 33, 1319–1334. doi:10.1139/H08-130
Burke, L.M. (2017). Practical issues in evidence-based use of performance supplements: Supplement interactions, repeated use and individual responses. Sports Medicine, 47(Suppl. 1), 79–100. doi:10.1007/s40279-017-0687-1
Burke, L.M., & Cato, L. (2015). Dietary supplements and nutritional ergogenic aids. In: L.M. Burke& V. Deakin (Eds.), Clinical sports nutrition (5th ed., pp. 493–582). Sydney, Australia: McGraw-Hill.
Carr, A.J., Gore, C.J., & Dawson, B. (2011a). Induced alkalosis and caffeine supplementation: Effects on 2,000-m rowing performance. International Journal of Sport Nutrition and Exercise Metabolism, 21, 357–364. doi:10.1123/ijsnem.21.5.357
Carr, A.J., Hopkins, W.G., & Gore, C.J. (2011b). Effects of acute alkalosis and acidosis on performance: A meta-analysis. Sports Medicine, 41, 801–814. doi:10.2165/11591440-000000000-00000
Carr, A.J., Slater, G.J., Gore, C.J., Dawson, B., & Burke, L.M. (2012). Reliability and effect of sodium bicarbonate: Buffering and 2000-m m rowing performance. International Journal of Sports Physiology and Performance, 7, 152–160. PubMed doi:10.1123/ijspp.7.2.152
Casey, A., & Greenhaff, P.L. (2000). Does dietary creatine supplementation play a role in skeletal muscle metabolism and performance? The American Journal of Clinical Nutrition, 72(Suppl. 2), 607S–617S.
Conger, S.A., Warren, G.L., Hardy, M.A., & Millard-Stafford, M.L. (2011). Does caffeine added to carbohydrate provide additional ergogenic benefit for endurance? International Journal of Sport Nutrition and Exercise Metabolism, 21, 71–84. PubMed doi:10.1123/ijsnem.21.1.71
Cox, G.R., Desbrow, B., Montgomery, P.G., Anderson, M.E., Bruce, C.R., Macrides, T.A., … Burke, L.M. (2002). Effect of different protocols of caffeine intake on metabolism and endurance performance. Journal Applied Physiology, 93, 990–999. doi:10.1152/japplphysiol.00249.2002
Currell, K., & Jeukendrup, A.E. (2008). Validity, reliability and sensitivity of measures of sporting performance sporting. Sports Medicine, 38, 297–316. PubMed doi:10.2165/00007256-200838040-00003
Del Coso, J., Pérez-López, A., Abian-Vicen, J., Salinero, J.J., Lara, B., & Valadés, D. (2014). Enhancing physical performance in male volleyball players with a caffeimne-containing energy drink. International Journal of Sports Physiology and Performance, 9, 1013–1018. doi:10.1123/ijspp.2013-0448
Del Coso, J., Portillo, J., Muñoz, G., Abián-Vicén, J., Gonzalez-Millán, C., & Muñoz-Guerra, J. (2013). Caffeine-containing energy drink improves sprint performance during an international rugby sevens competition. Amino Acids, 44, 1511–1519. PubMed doi:10.1007/s00726-013-1473-5
Del Coso, J., Portillo, J., Salinero, J.J., Lara, B., Abian-Vicen, J., & Areces, F. (2016). Caffeinated energy drinks improve high-speed running in elite field hockey players. International Journal of Sport Nutrition and Exercise Metabolism, 26, 26–32. PubMed doi:10.1123/ijsnem.2015-0128
Del Coso, J., Ramírez, J.A., Muñoz, G., Portillo, J., Gonzalez-Millán, C., Muñoz, V., … Muñoz-Guerra, J. (2013). Caffeine-containing energy drink improves physical performance of elite rugby players during a simulated match. Applied Physiology Nutrition and Metabolism, 38, 368–374. doi:10.1139/apnm-2012-0339
Drummond, G.B., & Tom, B.D. (2011). Presenting data: Can you follow a recipe? Journal of Physiology, 589(Pt 21):5007–5011. doi:10.1113/jphysiol.2011.221093
Erdman, K.A., Fung, T.S., Doyle-Baker, P.K., Verhoef, M.J., & Reimer, R.A. (2007). Dietary supplementation of high-performance Canadian athletes by age and gender. Clinical Journal of Sport Medicine, 17, 458–464. doi:10.1097/JSM.0b013e31815aed33
Felippe, L.C., Lopes-Silva, J.P., Bertuzzi, R., McGinley, C., & Lima-Silva, A.E. (2016). Separate and combined effects of caffeine and sodium-bicarbonate intake on judo performance. International Journal of Sports Physiology and Performance, 11, 221–226. doi:10.1123/ijspp.2015-0020
Glaister, M., & Gissane, C. (2017). Caffeine and physiological responses to submaximal exercise: A meta-analysis. International Journal of Sports Physiology and Performance, 5, 1–23. doi:10.1123/ijspp.2017-0312
Graham, T.E., & Spriet, L.L. (1991). Performance and metabolic responses to a high caffeine dose during prolonged exercise. Journal of Applied Physiology, 71, 2292–2298. PubMed doi:10.1152/jappl.1991.71.6.2292
Halperin, I., Pyne, D.B., & Martin, D.T. (2015). Threats to internal validity in exercise science: A review of overlooked confounding variables. International Journal of Sports Physiology and Performance, 10, 823–829. PubMed doi:10.1123/ijspp.2014-0566
Hobson, R.M., Saunders, B., Ball, G., Harris, R.C., & Sale, C. (2012). Effects of β-alanine supplementation on exercise performance: A meta-analysis. Amino Acids, 43, 25–37. PubMed doi:10.1007/s00726-011-1200-z
Hoon, M.W., Hopkins, W.G., Jones, A.M., Martin, D.T., Halson, S.L., West, N.P., … Burke, L.M. (2014). Nitrate supplementation and high-intensity performance in competitive cyclists. Applied Physiology Nutrition and Metabolism, 39, 1043–1049. doi:10.1139/apnm-2013-0574
Hoon, M.W., Johnson, N.A., Chapman, P.G., & Burke, L.M. (2013). The effect of nitrate supplementation on exercise performance in healthy individuals: A systematic review and meta-analysis. International Journal of Sport Nutrition and Exercise Metabolism, 23, 522–532. PubMed doi:10.1123/ijsnem.23.5.522
Hopkins, W.G., Hawley, J.A., & Burke, L.M. (1999). Design and analysis of research on sport performance enhancement. Medicine & Science in Sports & Exercise, 31, 472–485. PubMed doi:10.1097/00005768-199903000-00018
Hultström, M., Amorim de Paula, C., Antônio Peliky Fontes, M., Porcelli, S., Bellistri, G., Pugliese, L., … Rehman, S. (2015). Commentaries on viewpoint: Can elite athletes benefit from dietary nitrate supplementation? Journal of Applied Physiology, 119, 762–769. doi:10.1152/japplphysiol.00640.2015
Ioannidis, J.P., & Lau, J. (1999). Pooling research results: Benefits and limitations of meta-analysis. The Joint Commission Journal on Quality Improvement, 25, 462–469. PubMed doi:10.1016/S1070-3241(16)30460-6
Irwin, C., Desbrow, B., Ellis, A., O’Keeffe, B., Grant, G., & Leveritt, M. (2011). Caffeine withdrawal and high-intensity endurance cycling performance. Journal of Sports Science. 29, 509–515. doi:10.1080/02640414.2010.541480
Jeacocke, N.A., & Burke, L.M. (2010). Methods to standardize dietary intake before performance testing. International Journal of Sport Nutrition and Exercise Metabolism, 20, 87–103. PubMed doi:10.1123/ijsnem.20.2.87
Jones, A.M. (2014). Dietary nitrate supplementation and exercise performance. Sports Medicine, 44(Suppl. 1), 35–45. doi:10.1007/s40279-014-0149-y
Jonvik, K.L., Nyakayiru, J., van Loon, L.J.C., & Verdijk, L.B. (2015). Viewpoint: Can elite athletes benefit from dietary nitrate supplementation? Journal of Applied Physiology, 119, 759–761. PubMed doi:10.1152/japplphysiol.00232.2015
Joyce, S., Minahan, C., Anderson, M., & Osborne, M. (2012). Acute and chronic loading of sodium bicarbonate in highly trained swimmers. European Journal of Applied Physiology, 112, 461–469. doi:10.1007/s00421-011-1995-z
Lane, S.C., Hawley, J.A., Desbrow, B., Jones, A.M., Blackwell, J.R., Ross, M.L., … Burke, L.M. (2014). Single and combined effects of beetroot juice and caffeine supplementation on cycling time trial performance. Applied Physiology Nutrition and Metabolism, 39, 1050–1057. doi:10.1139/apnm-2013-0336
Malcata, R.M., Hopkins, W.G. (2014). Variability of competitive performance of elite athletes: A systematic review. Sports Medicine, 44, 1763–1774. PubMed doi:10.1007/s40279-014-0239-x
Maughan, R.J., Shirreffs, S.M., & Vernec,A. (2018). Making decisions about supplement use. International Journal of Sports Nutrition and Exercise Metabolism, 28. doi:10.1123/ijsnem.2018-0009
McDonald, S., Quinn, F., Vieira, R., O’Brien, N., White, M., Johnston, D., & Sniehotta, F.F. (2017). The state of the art and future opportunities for using longitudinal n-of-1 methods in health behaviour research: A systematic literature overview. Health Psychology Review, 13, 1–17.
McMahon, N.F., Leveritt, M.D., & Pavey, T.G. (2017). The effect of dietary nitrate supplementation on endurance exercise performance in healthy adults: A systematic review and meta-analysis. Sports Medicine, 47, 735–756. PubMed doi:10.1007/s40279-016-0617-7
Mengersen, K.L., Drovandi, C.C., Robert, C.P., Pyne, D.B., & Gore, C.J. (2016). Bayesian estimation of small effects in exercise and sports science. PLoS ONE, 11(4), 0147311. PubMed doi:10.1371/journal.pone.0147311
Mueller, S.M., Gehrig, S.M., Frese, S., Wagner, C.A., Boutellier, U., & Toigo, M. (2013). Multiday acute sodium bicarbonate intake improves endurance capacity and reduces acidosis in men. Journal of the International Society of Sports Nutrition, 10, 16. doi:10.1186/1550-2783-10-16
Mujika, I., & Padilla, S. (1997). Creatine supplementation as an ergogenic aid for sports performance in highly trained athletes: A critical review. International Journal of Sports Medicine, 18, 491–496. PubMed doi:10.1055/s-2007-972670
Peeling, P., Cox, G.R., Bullock, N., & Burke, L.M. (2015). Beetroot juice improves on-water 500 m time-trial performance, and laboratory-based paddling economy in national and international-level kayak athletes. International Journal of Sport Nutrition and Exercise Metabolism, 25, 278–284. PubMed doi:10.1123/ijsnem.2014-0110
Pérez-López, A., Salinero, J.J., Abian-Vicen, J., Valadés, D., Lara, B., Hernandez, C., … Del Coso, J. (2015). Caffeinated energy drinks improve volleyball performance in elite female players. Medicine & Science in Sports & Exercise, 47, 850–856. PubMed doi:10.1249/MSS.0000000000000455
Pickering, C., & Kiely, J. (2017). Are the current guidelines on caffeine use in sport optimal for everyone? Inter-individual variation in caffeine ergogenicity, and a move towards personalised sports nutrition. Sports Medicine, 48(1), 7–16. doi:10.1007/s40279-017-0776-1
Porcelli, S., Ramaglia, M., Bellistri, G., Pavei, G., Pugliese, L., Montorsi, M., … Martorati, M. (2015). Aerobic fitness affects the exercise performance responses to nitrate supplementation. Medicine & Science in Sports & Exercise, 47, 1643–1651. PubMed doi:10.1249/MSS.0000000000000577
Portillo, J., Del Coso, J., & Abián-Vicén, J. (2017). Effects of caffeine ingestion on skill performance during an international female rugby sevens competition. Journal of Strength and Conditioning Research, 31(12), 3351–3357. doi:10.1519/JSC.0000000000001763
Rhodes, K., & Braakhuis, A. (2017). Performance and side effects of supplementation with N-Acetylcysteine: A systematic review and meta-analysis. Sports Medicine, 47(8):1619–1636. PubMed doi:10.1007/s40279-017-0677-3
Saunders, B., Elliott-Sale, K., Artioli, G.G., Swinton, P.A., Dolan, E., Roschel, H., … Gualano, B. (2016). β-alanine supplementation to improve exercise capacity and performance: A systematic review and meta-analysis. British Journal of Sports Medicine, 51, 658–669. doi:10.1136/bjsports-2016-096396
Shaw, G., Slater, G., & Burke, L.M. (2016). Supplement use of elite Australian swimmers. International Journal of Sport Nutrition and Exercise Metabolism, 26, 249–258. PubMed doi:10.1123/ijsnem.2015-0182
Somerville, V., Bringans, C., & Braakhuis, A. (2017). Polyphenols and performance: A systematic review and meta-analysis. Sports Medicine, 47, 1589–1599. doi:10.1007/s40279-017-0675-5
Souza, D.B., Del Coso, J., Casonatto, J., & Polito, M.D. (2017). Acute effects of caffeine-containing energy drinks on physical performance: A systematic review and meta-analysis. European Journal of Nutrition, 56, 13–27. PubMed doi:10.1007/s00394-016-1331-9
Stadheim, H.K., Spencer, M., Olsen, R., & Jensen, J. (2014). Caffeine and performance over consecutive days of simulated competition. Medicine and Science in Sports and Exercise, 46, 1787–1796. PubMed doi:10.1249/MSS.0000000000000288
Stellingwerff, T., & Cox, G.R. (2014). Systematic review: Carbohydrate supplementation on exercise performance or capacity of varying durations. Applied Physiology Nutrition and Metabolism, 39, 998–1011. doi:10.1139/apnm-2014-0027
Talanian, J.L., & Spriet, L.L. (2016). Low and moderate doses of caffeine late in exercise improve performance in trained cyclists. Applied Physiology Nutrition and Metabolism, 41, 850–855. doi:10.1139/apnm-2016-0053
Thomas, D.T., Erdman, K.A., & Burke, L.M. (2016). Position of the Academy of Nutrition and Dietetics, Dietitians of Canada, and the American College of Sports Medicine: Nutrition and athletic performance. Journal of the Academy of Nutrition and Dietetics, 116, 501–528. PubMed doi:10.1016/j.jand.2015.12.006
Tobias, G., Benatti, F.B., De Salles Painelli, V., Roschel, H., Gualano, B., Sale, C., … Artioli, G.G. (2013). Additive effects of β-alanine and sodium bicarbonate on upper-body intermittent performance. Amino Acids, 45, 309–317. doi:10.1007/s00726-013-1495-z
Watt, K.K., Garnham, A.P., & Snow, R.J. (2004). Skeletal muscle total creatine content and creatine transporter gene expression in vegetarians prior to and following creatine supplementation. International Journal of Sport Nutrition and Exercise Metabolism, 14, 517–531. doi:10.1123/ijsnem.14.5.517
Welsh, A.H., & Knight, E.J. (2015). “Magnitude-based inference”: A statistical review. Medicine & Science in Sports & Exercise, 47, 874–884. PubMed doi:10.1249/MSS.0000000000000451
Weissgerber, T.L., Milic, N.M., Winham, S.J., & Garovic, V.D. (2015). Beyond bar and line graphs: Time for a new data presentation paradigm. PLoS Biology, 13(4), e1002128. PubMed doi:10.1371/journal.pbio.1002128
Womack, C.J., Saunders, M.J., Bechtel, M.K., Bolton, D.J., Martin, M., Luden, N.D., … Hancock, M. (2012). The influence of a CYP1A2 polymorphism on the ergogenic effects of caffeine. Journal of the International Society of Sports Nutrition, 9, 7. PubMed doi:10.1186/1550-2783-9-7
Womack, C.J., Saunders, M.J., Bechtel, M.K., Bolton, D.J., Martin, M., Luden, N.D., … Hancock, M. (2015). Erratum to: The influence of a CYP1A2 polymorphism on the ergogenic effects of caffeine. Journal of the International Society of Sports Nutrition, 12, 24. PubMed doi:10.1186/s12970-015-0079-6
Yang, A., Palmer, A.A., & de Wit, H. (2010). Genetics of caffeine consumption and responses to caffeine. Psychopharmacology, 211(3), 245–257. PubMed doi:10.1007/s00213-010-1900-1
