Over the past 40 years, published reviews on physical activity and health-related outcomes have proliferated. Indeed, we conducted searches in PubMed with terms related to the various types of reviews on February 24, 2021 and found 76,768 “reviews on physical activity” from 1843 to 2021 (Figure 1); 6076 “systematic reviews (SRs) and meta-analyses (MAs) on physical activity” from 1996 to 2021; 3235 “overviews of reviews on physical activity” from 1976 to 2021; 1853 “narrative reviews on physical activity” from 1992 to 2021; 178 “umbrella reviews on physical activity” from 1995 to 2021; 58 “meta-syntheses on physical activity” from 2005 to 2021; and 4 “meta-reviews (MRs) on physical activity” from 2004 to 2021. Due to their importance, there is a need to translate review findings into the practice and policy base as quickly as possible. Yet, given the rapid growth of users of this method, there are a variety of methods employed, resulting in disparate conclusions across reviews.1,2 Reviews of reviews, or MRs, have become an increasingly popular way to address the sheer amount of information generated and make sense of review findings.I
—Number of reviews on physical activity by publication year in PubMed (Accessed 02/24/2021).
Citation: Journal of Physical Activity and Health 18, 11; 10.1123/jpah.2021-0243
—Number of reviews on physical activity by publication year in PubMed (Accessed 02/24/2021).
Citation: Journal of Physical Activity and Health 18, 11; 10.1123/jpah.2021-0243
—Number of reviews on physical activity by publication year in PubMed (Accessed 02/24/2021).
Citation: Journal of Physical Activity and Health 18, 11; 10.1123/jpah.2021-0243
Most journals related to physical activity and health now recommend authors refer to the Preferred Reporting Items for SRs and MAs (PRISMA) when preparing their reviews for publication.3,4 PRISMA discourages narrative reviews and at the same time encourages well-conducted SRs/MAs. Well-conducted SRs/MAs adhering to high-quality, standardized publication reporting guidelines such as PRISMA are important because they often provide the scientific foundation for position stances, scientific statements and reports, and public health recommendations and policy.2,5,6 With the emergence of MRs, defined as a systematic review of SRs/MAs that may or may not be quantified, adherence to high-quality, contemporary methodological standards at each level of review is imperative. In theory, due to their scope and impact, MRs should provide the strongest level of evidence of all types of reviews.7,8 Table 1 describes the various types of reviews and recommended citation sources.1,8–15 Presently, practical guidance on how to conduct MRs is lacking in the physical activity and health literature; we aim to fill this gap with this article.
The Hierarchy of the Common Types of Reviews (Evidence Syntheses)
| Type of review | Definition |
|---|---|
| Meta-review8,9,24,29,33,34,41 | Gathers systematic reviews, meta-analyses, and/or meta-syntheses and provides qualitative, semi-quantitative, or quantitative conclusions. Summarizes critically appraised, evaluated, and synthesized research questions(s) relating to study-level evidence using and documenting comprehensive, systematic search strategies, and rigorous, transparent appraisal methods. May review both qualitative and quantitative evidence on the topic. |
| Umbrella review7 | Like the meta-review, an umbrella review gathers systematic reviews and/or meta-analyses, but it also collects and incorporates primary studies. Often, the primary studies are included to account for newer literature or underdeveloped evidence syntheses in a particular subset of research that is of interest to the overall review. Umbrella review methods typically mirror those of the meta-review. |
| Scoping review13 | A systematic synthesis of available evidence on a subject. Emphasis is more on gauging what literature exists than in evaluating conclusions reached in the literature. Can evaluate both primary- (ie, original research studies on individuals) and secondary-level sources (ie, study-level evidence from systematic reviews and meta-analyses). |
| Meta-analysis14,15 | Quantifies and combines the findings of independent research studies. Calculates overall summary statistics of individual effect sizes across studies. Effect sizes quantify the strength of association between 2 variables or the magnitude of a variable present. Often examine effect modifiers (moderators) effect sizes. Meta-analyses can be conducted at the study or aggregate level or individual participant level, the latter of which obtains the actual participant data. In practice, the term meta-analysis is often used to imply systematic review, even though this statistical procedure may be based on any subset of studies (ie, meta-analyses are not necessarily based on a systematic review of the literature). |
| Systematic review1,2,10 | Summarizes critically appraised, evaluated, and synthesized research questions(s) relating to individual research studies using and documenting comprehensive search strategies and rigorous, transparent appraisal methods. |
| Meta-synthesis11 | Systematically pools qualitative evidence from multiple studies on a given topic and forms conclusions about emergent trends. (More rarely, is a synthesis of effect sizes pooled from multiple syntheses. Meta-syntheses may also be combined or mixed with quantitative syntheses.) |
| Narrative review12 | A summary of evidence on a given topic, often with the goal of providing a broad perspective on a topic (eg, like a textbook chapter). Tends to rely on expert opinion and subjective knowledge of the extant literature. Not regarded as a rigorous strategy of evidence synthesis, although can be a source of possible insights into a given phenomenon. |
We begin with a brief overview of the methodology of the 2018 Physical Activity Guidelines Advisory Committee (PAGAC) Scientific Report (PAGACSR),16 which served as the scientific foundation for the Physical Activity Guidelines for Americans, second edition (PAG).17 The PAG is an important public policy document guiding children and adults on ways to improve health through a physically active lifestyle. The PAGAC encountered several challenges when performing their MRs for the 2018 PAGACSR. In turn, these are important lessons for scientists, clinicians, and researchers working in the field of physical activity and health and should be shared. Thus, we provide (1) an overview of the 2018 PAGACSR methodology; (2) a high-level introduction to the methodology of the MR; (3) examples of the challenges the PAGAC encountered when conducting their MRs; and (4) best practice guidance in addressing these challenges. We conclude with implications for the future direction of physical activity and health research and evidence synthesis.
2018 PAGACSR Methodology
Due to substantial growth of the literature since the publication of the 2008 PAGACSR,16,18 the 2018 PAGAC set out to conduct a MR of SRs/MAs and pooled analyses to answer 38 primary research questions (with 104 secondary questions). This MR was split among 9 subcommittee teams such that those with the most substantive expertise in a particular subdomain synthesized findings from that area.16,18 One challenge the PAGAC encountered was several of the initial literature searches identified only a limited number of SRs/MAs (or none). The PAGAC then performed a SR of the primary literature. The sources of evidence could then include (1) a MR of SRs/MAs and pooled analyses; (2) a SR of original research; or (3) a combination of both. In all instances, the methodology adhered to the best practices of SRs consistent with PRISMA,3,4 the United States Department of Agriculture (USDA) Nutrition Evidence-based Library (NEL),19 the Agency for Healthcare Research and Quality,20 the Cochrane Collaboration,21 and the 2011 Institute of Medicine Systematic Review Standard.22
The SR methods are described in detail elsewhere5,16,18 and are briefly overviewed here: All protocols were registered in the International Prospective Register of Systematic Reviews (PROSPERO); there was an extensive search for relevant literature across multiple electronic databases and through expert contact; all data extraction was done independently and in duplicate; tools appropriate to the study design were used to systematically assess study quality (ie, Assessment of Multiple Systematic Reviews [AMSTAR]23 adapted for physical activity and health outcomes [AMSTARExBP]6 for SRs/MAs, and the USDA NEL Bias Assessment Tool19 for original research); and the outcomes were graded based on the strength of the evidence identified.19
In the sections that follow, representing authors of the PAGAC Sedentary Behavior (P.T.K.), Cardiometabolic Health and Prevention of Weight Gain and Chronic Conditions (W.E.K. and L.S.P.), and Exposure Subcommittees (W.E.K. and L.S.P.) provide examples of the obstacles and challenges encountered by their subcommittees and the solutions used to overcome them. The entire process is detailed for each research question in an evidence portfolio available at PAGACSR (https://health.gov/our-work/physical-activity).16
Guidance in Best Practices for MRs on Physical Activity and Health
Conventions about best methods for conducting MRs are only now emerging. Nonetheless, as a starting point, they should follow methods established as best practice for SRs, coupled with the methods to address the specific issues encountered when conducting a MR: methods to gauge the quality, datedness, and overlap of sampled evidence syntheses. Cooper and Koenka24 provided the first formal overview of MR methods, and Hennessy et al8 refined them for use in the area of health psychology. The advice in these sources can be divided into the 7 interrelated steps summarized in Figure 2.7–9,24–30 The following sections detail important considerations within each of these steps that are specific to physical activity and health research. We also provide examples from the 2018 PAGACSR experience of conducting a large-scale MR in these steps. Before undertaking these 7 steps, preregistration of the MR, as was done for all PAGACSR protocols in PROSPERO, is another marker of quality, as it prespecifies the hypotheses, methods, and planned analyses; preregistration also minimizes the redundancies that can occur when applying the 7 steps that we now describe.1,2,8
—The 7 best practice steps for meta-reviews. The right-pointing smaller lower arrows portray how details of one step lead to the next, creating synergies across the steps. The left-pointing larger upper arrows model show how information learned at any step can cycle a meta-review team to return to earlier steps, then proceed again through the remaining steps. This figure was drawn by and is the courtesy of Daniela Inge Sage Johnson.
Citation: Journal of Physical Activity and Health 18, 11; 10.1123/jpah.2021-0243
—The 7 best practice steps for meta-reviews. The right-pointing smaller lower arrows portray how details of one step lead to the next, creating synergies across the steps. The left-pointing larger upper arrows model show how information learned at any step can cycle a meta-review team to return to earlier steps, then proceed again through the remaining steps. This figure was drawn by and is the courtesy of Daniela Inge Sage Johnson.
Citation: Journal of Physical Activity and Health 18, 11; 10.1123/jpah.2021-0243
—The 7 best practice steps for meta-reviews. The right-pointing smaller lower arrows portray how details of one step lead to the next, creating synergies across the steps. The left-pointing larger upper arrows model show how information learned at any step can cycle a meta-review team to return to earlier steps, then proceed again through the remaining steps. This figure was drawn by and is the courtesy of Daniela Inge Sage Johnson.
Citation: Journal of Physical Activity and Health 18, 11; 10.1123/jpah.2021-0243
Define the Scope of the Meta-Review
Like all research endeavors, a question must first be posed. As Hennessy et al8 argued, there are 2 reasons why this step is the most important of the entire evidence synthesis process. First, a practical consideration: defining the question has direct ramifications for how large the project will be, and thus, what resources are necessary to complete it efficiently. It is important at this stage to conduct several preliminary literature searches to ensure the appropriate terms are included and get a sense of the size of the literature base. These informal searches often result in tightening (or broadening) the scope of the project.
Given the challenge of the sheer scope of the literature to review (Figure 1 displays physical activity and health literature from only one electronic database), and given the short amount of time the PAGAC had to complete the review (1 y), several inclusion/exclusion criteria were used to limit the scope and make the article search and retrieval process feasible within the time frame allocated. Articles were initially limited to peer-reviewed SRs/MAs and pooled analysis published in English with human subjects. If the retrieved SRs/MAs did not provide enough information to address the research questions (a challenge encountered by some of the PAGAC subcommittees), then a SR of original research studies was conducted. When necessary, the use of this combined approach of identifying relevant SRs/MAs, extracting the results from studies contained in those SRs, and performing an updated SR of more recent original articles allowed the PAGAC to get a fuller understanding of the evidence base for the associations between physical activity and health they were examining.
Second, the question of scope shapes what happens in all the subsequent steps of conducting a MR. Research findings in primary-level studies, both in physical activity and health research and more generally across all sciences, are routinely heterogeneous,25–27 thus the findings vary more than sampling error along suggests they should. Accordingly, SRs/MAs routinely investigate a priori hypotheses addressable with effect modifier (moderator) analyses. In defining the scope of the MR, the evidence synthesis team should consider what independent and dependent variables are most relevant, and what moderator variables are of most interest. Reviewers often use a research question framework such as PICO (Population, Intervention/Exposure, Comparison, and Outcome)II to ensure that all aspects of the inclusion criteria match the primary aims of the project.
To represent the PICO questions formulated by the PAGAC, an analytic framework was created for each of the 38 research questions in the PAGACSR.16,18 See Table 2 for an example of the research question framework created by the Cardiometabolic Health and Prevention of Weight Gain and Chronic Conditions subcommittees on physical activity and blood pressure. The analytical frameworks established the foundation for the search strategies, inclusion/exclusion criteria, and search terms, and was also used to refine the research questions if needed. Another example was the research question addressed by the PAGAC Sedentary Behavior Subcommittee: What is the relationship between sedentary behavior and all-cause mortality? This question was framed by 4 sub-research questions important to understand the relationship between sedentary behavior and health: (1) Is there a dose–response relationship? If yes, what is the shape of the relationship? (2) Does the relationship vary by age, sex, race/ethnicity, socioeconomic status, or weight status? (3) Is the relationship independent of amounts of light, moderate, or vigorous physical activity? and (4) Is there any evidence that bouts or breaks in sedentary behavior are important factors?
The Research Questions Related to the Relationship Between Physical Activity and BP Among Adults With Normal BP, Prehypertension, or Hypertension Addressed by the 2018 Physical Activity Guidelines Advisory Committee16,32
| Major questions |
|---|
| (1) In people with normal BP, prehypertension, or hypertension, what is the relationship between physical activity and BP? (2) In people with the cardiovascular condition of hypertension, how does physical activity associate with: (a) risk of co-morbid conditions, (b) physical function, (c) health-related quality of life, and (d) cardiovascular disease progression and mortality? |
| Sub-questions |
|---|
| (1) Is there a dose–response relationship? If yes, what is the shape of the relationship? (2) Does the relationship vary by age, sex, race/ethnicity, socioeconomic status, weight status, or resting BP level? (3) Does the relationship vary based on frequency, duration, intensity, type (mode), or how physical activity is measured? |
Abbreviations: BP, blood pressure; DBP, diastolic BP; JNC, Joint National Commission; SBP, systolic BP. Notes: The 2018 Physical Activity Guidelines Advisory Committee used the JNC 7 BP classification scheme for data extraction purposes. The JNC 7 defines these BP classifications as follows: Hypertension is defined as having a resting SBP of ≥ 140 mm Hg and/or a resting DBP of ≥ 90 mm Hg, or taking antihypertensive medication, independent of the resting BP level. Prehypertension is defined as a resting SBP of 120 to 139 mm Hg and/or DBP of 80 to 89 mm Hg. Normal BP is defined as a resting SBP < 120 mm Hg and DBP < 80 mm Hg.
Balancing the challenge of the size of the literature to be searched and the degree of overlap in the original research articles in the retrieved SR/MAs, the Cardiometabolic Health and Prevention of Weight Gain and Chronic Conditions subcommittees initial search of SRs/MAs and pooled analyses was sufficient to answer the research questions on the relationship between physical activity and blood pressure (Table 2). However, the PAGAC Sedentary Behavior subcommittee needed a combined approach to obtain a fuller understanding of the evidence base for the association between sedentary behavior and all-cause mortality. All searches in the PAGACSR were limited to the English language. Although English has been a favored scientific language for several decades, it has not always been. Scholars who lack English-language fluency often pursue publication in non-English sources because non-English journals may be more tailored to addressing local or national issues pertinent to their research, and there are a limited number of English journals that provide adequate editing of international submissions, among others. Evidence syntheses focusing only on English-language sources risk creating a mono-language bias, such that results cannot be explicitly generalized to other, potentially more diverse contexts or populations. Many evidence syntheses are not published in peer-reviewed outlets and instead wind up in theses, dissertations, and other outlets, including professional websites that are not indexed/searched by electronic databases/hosts, termed gray literature. Thus, if resources allow, gray literature and literature in other languages should be gathered and coded for potential methodological strengths and limitations. In this way, evidence from differing literature sources can be compared via sensitivity analyses to see if the same conclusions are reached or if there is a higher likelihood of reporting bias in the literature being reviewed.
Conduct the Formal Literature Search
The goal of this step is to find all evidence syntheses addressing the MR team’s research question(s). Reporting standards such as PRISMA3,4 and methodological quality inventories such as AMSTAR,23 which are hallmarks of a high-quality SRs, recommend (1) selection of multiple databases; (2) use of appropriately broad search terms; (3) retrieval of gray literature; (4) no language limitation; and (5) all with guidance from a librarian who specializes in the field of inquiry. As described earlier, the scope of the project will drive the extent to which each of these recommendations is followed.
Although considerable overlap exists, multiple databases are necessary because no database has documented all scientific literature on a question. Conventionally, 2 databases are lauded as “good enough”23; but in practice, evidence syntheses routinely search many more. The point is to find all reputable evidence syntheses; thus, using more databases is good practice within the resources available to the evidence synthesis team. The searches in the 2018 PAGACSR were conducted by a consulting services company in the 3 electronic databases of PubMed®, Cumulative Index to Nursing and Allied Health Literature, and the Cochrane library. These searches were supplemented by the PAGAC subject matter experts to identify additional articles relevant to the search. In total, the PAGAC performed 38 literature searches across these 3 electronic databases and triaged 20,838 titles, 4913 abstracts, and 2139 full texts of potentially qualifying articles from which 1130 articles qualified, and were abstracted and assessed.
The search terms for the database searches need to be specified with sufficient breadth to capture the research question but not so broad that many false positives are returned, making it unwieldy to screen reports for inclusion. Fortunately, searches for evidence syntheses can, and usually should be, piloted and refined using Boolean terms (ie, AND, OR, NOT to limit) to help reduce the number of reports and increase the search’s yield. The main advantage in conducting a MR search is the synonyms for “evidence synthesis” in its many forms are quite well known. When these MR synonyms are included in the search, as was done in the PAGACSR, there are far fewer potentially qualifying reports to screen.5,28
Due to the complex nuances of concerning searches, high expertise is necessary to complete high-quality literature reviews.29 Hence, librarians or experts with appropriate specialization are routinely included in the process and often become authors of MRs.5,30–32 When experts are not included in the search methods, investigators risk being naïve to their own deficiencies at searching for literature, acting with over-confidence, and conducting searches that are far from complete.2 Librarians and experts with appropriate specialization, such as the consulting firm that performed the PAGACSR search, in contrast, maintain contemporary knowledge of database limitations as well as the skills needed to search them. Although including experts may increase the time necessary to do a thorough literature search, it can often decrease the time needed to screen reports for inclusion.
Extract Data and Code Reviews (for Quality and Other Relevant Variables)
As with other aspects of the MR process, best practices in evidence synthesis are standard across review types.29,33,34 Data should be extracted independently and in duplicate using a piloted and standardized data collection form. It is only by following this process that readers are assured the conclusions made in the synthesis are trustworthy.24,35
The major research questions and sub-questions addressed by the PAGAC Cardiometabolic Health and Prevention of Weight Gain and Chronic Conditions subcommittees on physical activity and blood pressure among adults with normal blood pressure, prehypertension, and hypertension are shown in Table 2. Due to the challenge of the lack of reporting of the frequency, intensity, and time (ie, FIT) of the physical activity exposure, there was insufficient evidence to determine whether there was a dose–response relationship between physical activity volume and blood pressure among adults with prehypertension and hypertension. Furthermore, even when the combined search approach was used by the PAGAC Sedentary Behavior subcommittee, they still concluded there was limited evidence that the relationship between sedentary behavior and all-cause mortality varied by age, sex, race/ethnicity, socioeconomic status, or weight status. Unfortunately, given the level of evidence and/or the quality of reporting in the included SRs/MAs, much of the evidence to address the PAGAC’s effect modifier questions was graded as “not assignable.”16,30–32
Alternatively, the combined search approach better informed the PAGAC Sedentary Behavior subcommittee on the relationship between sedentary behavior and all-cause mortality because they found 2 MAs demonstrating the existence of a dose–response relationship.36,37 The 2 MAs were supplemented by evidence from 24 out of 29 original research studies that also found significant dose–response associations between sedentary behavior and all-cause mortality.16,30 Furthermore, the PAGAC Cardiometabolic Health and Prevention of Weight Gain and Chronic Conditions subcommittees concluded in their MR due to strong evidence that physical activity reduces blood pressure among adults with normal BP, prehypertension, and hypertension, and the greatest blood pressure reductions occur among adults with hypertension.16,32 This finding was consistent with the law of initial values38; those with the highest resting blood pressure levels, experienced the greatest blood pressure benefits from physical activity.
Because the PAGAC used a standardized way of collecting and synthesizing these data, they were able to present findings relevant to practice and policy in the PAG,17 as well as to suggest recommendations for future primary research in the PAGACSR.16 Evidence syntheses deviating from contemporary standards may well reach different conclusions than those that follow rigorous standards. Thus, obtained evidence syntheses should be coded for their degree of rigor. Accordingly, the PAGAC applied the AMSTARExBP6 for SRs/MAs and the USDA NEL Bias Assessment Tool19 for original research to judge whether relevant conclusions are robust to observed flaws, and the outcomes were graded based on the strength of the evidence identified.19 See Table 3 for the PAGACSR criteria for grading the strength of evidence.
The 2018 Physical Activity Guidelines Advisory Committee Criteria for Grading the Strength of Evidence
| Criteria | Strong | Moderate | Limited | Grade not assignable |
|---|---|---|---|---|
| A. Directness | Study populations, interventions, and outcomes are directly related to the systematic review question | Some of the study populations, interventions, and outcomes, are directly related to the systematic review question | Most of study populations, interventions, and outcomes relate to the systematic review question indirectly | All study populations, interventions, and outcomes relate to the systematic review question indirectly |
| B. Quantity | Several studies; large number of subjects studied, sufficiently large sample size for adequate statistical power | Several studies; doubts about adequacy of sample size to avoid Type I and Type II error | Limited number of studies; low number of studies and/or inadequate sample size within studies | No studies available that directly answer the question |
| C. Risk of bias/study limitations | Studies are of strong design; free from methodological concerns, bias, and execution problems | Studies are of strong design with minor methodological concerns OR studies of weaker study design | Studies of weak design OR inconclusive findings due to design flaws, bias, or execution problems | Serious design flaws, bias, or execution problems across the body of evidence |
| D. Consistency (of results across the available studies) | Results are generally consistent in direction, size of effect, degree of association, and statistical significance | Some inconsistency in results, direction, size of effect, degree of association, or statistical significance | Results are generally inconsistent in direction, size of effect, degree of association, and statistical significance | Findings are too disparate to synthesize OR single small study unconfirmed by other studies |
| E. Magnitude of effect | Size of effect is “practically” meaningful | Some doubt about the practical significance of the effect | Size of effect is small or lacks practical significance | Size of effect cannot be determined |
| F. Generalizability (across populations in the United States) | Studied population, intervention, and outcomes are free from serious doubts about generalizability | Minor doubts about generalizability | Serious doubts about generalizability due to narrow or different study population, intervention, or outcomes studied | Highly unlikely that the studied population, intervention, and/or outcomes are generalizable to the population of interest |
Weigh Overlap and Datedness
Evidence syntheses may reach different conclusions because they define the studied phenomenon differently. Thus, once the evidence syntheses are selected and coded in a MR, the next step is to examine their focal questions and methods. The MR team should inspect the databases searched, the search terms that individual evidence syntheses used, and the selection criteria applied to reports to assess the degree of overlap among included SRs/MAs. Depending on the degree of overlap and SR methods, SRs/MAs may include some (or all or none) of the same primary studies (known as overlap). An evidence synthesis that updates one conducted 5 years earlier theoretically would be expected to have more studies than the first if the studies continued to appear in the literature. Nonetheless, the PAGAC Cardiometabolic Health and Prevention of Weight Gain and Chronic Conditions subcommittees, when weighing the challenge of the 2 potentially qualifying MAs on the role of isometric resistance training to prevent and treat hypertension, chose the older of the two. The reasons for this choice were that there was significant overlap in the primary studies between the 2 MAs and the older MA had a higher methodological quality rating than the more recent MA.16,32,39 In a MR of health-related or intervention research such as the PAGACSR, it is important to ensure that the evidence synthesized accurately represents current practice, which both of these MAs did despite their significant overlap. It is also important to examine the years of research included in SRs/MAs to ensure the reviews synthesizing these studies have addressed any temporal trends that bear caution; none of which were found in the 2 MAs on isometric resistance training and blood pressure.
The PAGAC Sedentary Behavior subcommittee found it challenging to address their primary research question, “What is the relationship between sedentary behavior and all-cause mortality?” from the SRs/MAs available. After screening relevant results, only 9 SRs/MAs qualified. There was considerable overlap in the original research articles reviewed across the SRs/MAs, and there were insufficient details disclosed in the SRs/MAs to answer the sub-questions of interest. Therefore, the PAGAC Sedentary Behavior Subcommittee decided to (1) extract information from the 20 original research articles reviewed in the 9 identified SR/MAs and (2) conduct a SR of additional original prospective longitudinal research studies published since the SRs/MAs. The updated SR identified a total of 1214 articles, of which 27 were deemed relevant after review of titles, abstracts, and full texts.16,30 This combined approach addressed the issue of understanding the degree of overlap in primary studies across the SRs/MAs and allowed the PAGAC to better answer the research sub-questions.
Of interest, there are several ways to examine overlap between the evidence syntheses in the sample. For example, Pieper et al40 developed the corrected covered area index to quantify the degree of overlap between studies included in separate evidence theses. There are also several ways to address overlap, the choice of which will depend on the degree and source of overlap40; Hennessy and Johnson9 further clarified the interpretation of the corrected covered area index to incorporate considerations of methodological quality, datedness, and unexplained heterogeneity of findings.
Data Synthesis
Once the data are extracted and the review quality assessed, the meta-reviewer has the task of compiling all these findings in a way that will distill the most important findings for readers.33,41 Although this step comes later in the chain of methods described here as shown in Figure 2, MRs must plan for this step at the scoping level of the project. For example, the decision to include different study designs, such as observational and intervention research in the same MR, will dictate the reviews’ approach to how the data will be synthesized and reported. Due to the challenges of the size of the physical activity and health literature and the time constraints imposed on the PAGAC to complete their SR, the PAGACSR was not quantified. Rather, the 2018 PAGAC first carefully deliberated the qualifying reviews, and then applied a rubric adapted from the USDA19 to grade the evidence for the conclusion statements as strong, moderate, limited, or “not assignable” (Table 3). Grading the evidence for the posed research questions was based upon these 5 criteria: applicability, generalizability, risk of bias/study limitations, quantity and consistency of results across studies, and magnitude and precision of effect.
Disseminate the Findings
Interpretation of MRs results hinges on all the preceding steps. All methodological steps are reported precisely; aspects occurring after registration or because of learning during the process are reported as after the fact.33,41 Indeed, it is difficult to imagine how a MR can achieve high quality without the cross-step synergy Figure 2 depicts. Any trends in results across the syntheses in the MR that emerge are reported and conclusions reached. Ultimately, the MR team pursues dissemination of their work in a timely fashion, typically in peer-reviewed journals. However, to translate findings to practice, it may also be necessary to provide laypeople, providers, and policy makers with summaries tailored to their needs.
A final step may result in peer review after publication, such that a MR team returns to earlier steps to make changes and update the MR. Ideally, the team provided a license for their data, posted a data availability statement, and archived their materials on an open-access platform.42 Adherence to these practices, will allow for more efficient updating of the MR and availability to other researchers to assess the validity of claims made in the original MR.42 For example, the team should document the exact reasons for any reports excluded from the review (eg, in an appendix). This way, teams evaluating the MR’s conclusions can evaluate assumptions underlying the original team’s work.
The PAGAC findings were disseminated in several ways. First, the 2018 PAGACSR16 was the MR that provided the scientific foundation for the PAG,17 an important public policy document that guides children and adults on ways to improve health through a physically active lifestyle. The process is designed to ensure independent review in an open published manner with opportunities for public participation with multiple levels of peer review. The authorship teams of these 2 documents are separate and distinct from one another. The PAGACSR is written by the appointed Advisory Committee members that submit the PAGACSR to the Department of Health and Human Services Secretary for development of the policy document written by federal staff members from the Office of Disease Prevention and Health Promotion; Centers for Disease Control and Prevention; National Institutes of Health; and the President’s Council on Sports, Fitness, and Nutrition. Second, both documents are openly available for download from https://health.gov/our-work/physical-activity. Third, selected topics reviewed in the PAGACSR were published as a series of Pronouncements from the American College of Sports Medicine openly available for download from https://www.acsm.org/acsm-positions-policy/physical-activity-guidelines-for-americans, with those of Kraus et al,31 Katzmarzyk et al,30 and Pescatello et al32 included in this report. In addition, there have been ancillary publications summarizing both the methods and highlighting results from the PAGACSR including this manuscript.5,18,43–46
Re-Analysis, Development, or Criticism (Future MRs)
Dissemination of a single MR should not be considered the final word on the subject, as science demands replication and new evidence routinely accumulates, often hastened by prior SRs and MRs. Thus, it is best for MR teams to ease the work of future MR teams, which is a logical seventh step in the process. Accordingly, there are additional referred publications summarizing both the methods and highlighting results from the PAGACSR5,18,43–46; these sources suggest potential improvements for future science in this arena. We provide further suggestions in the next section.
Implications for the Future Directions in Physical Activity and Health Research
Any scientific method has potential limitations. Tacit in the foregoing discussion has been the assumption that a team conducts the MR, not an individual. In this way, there are checks and balances throughout the process regarding the methodology employed and appropriate content expertise of the team. Thus, at the onset, the scope of the project can be made more manageable and be performed in a reasonable time frame with efficient use of resources. Coding procedures may be refined to be more precise or inclusive. Reliability figures can be calculated, providing assurances that judgments of the literature can be trusted. The process may be highly educational to all members of the team, but perhaps especially for those who are undergoing formal training on the subject in review (eg, graduate students in kinesiology and public health).
We also recommend prospective MR teams first check whether relevant MRs already exist and whether their conclusions can be trusted. If the conclusions can be trusted, the need for a new MR is dubious except as an effort to replicate its conclusions. If the extant MRs can be trusted, their conclusions as to the best future research to conduct ought to be followed, ranging from new randomized controlled trial or cohort studies, to evidence syntheses focused on new research questions. Indeed, there is an entire section in the 2018 PAGACSR on needs for future research. Of course, scholars wishing to conduct new primary-level trials based on MRs should be alert to the assumptions MRs may make and adjust accordingly. For example, if MRs have not reviewed key relevant literature, it would seem more appropriate to conduct a new MR first.
Finally, producing a strong MR requires the formation of a MR team highly skilled in contemporary methods of evidence synthesis as outlined in this manuscript as well as content experts in the areas being reviewed and evaluated. The MR team should anticipate network MA, an increasingly popular and more complex form of analysis that involves strict assumptions around the transitivity of included groups/interventions47–52 to ensure direct and indirect comparisons can be used. Only someone versed in network MA will be able to address whether the conclusions reached are valid. Because the conclusions made by SRs/MAs are dependent upon the primary-level literature they review and evaluate, and considering the challenges encountered by the PAGAC in conducting their MR such as the general lack of disclosure of key effect modifiers in the primary-level studies, we urge investigators to adhere to the methodological guidance provided by such tools as the USDA NEL Bias Assessment Tool19 for original primary-level research.
Acknowledgments
The authors gratefully acknowledge the contributions of the 2018 PAGAC who were involved in the initial search of the secondary literature: David M. Buchner, Wayne Campbell, Loretta DiPietro, Kirk I. Erickson, Charles H. Hillman, John M. Jakicic, Kathleen F. Janz, Peter T. Katzmarzyk, Abby C. King, William E. Kraus, Richard F. Macko, David X. Marquez, Anne McTiernan, Russell R. Pate, Linda S. Pescatello, Kenneth E. Powell, and Melicia C. Whitt-Glover; Anne Brown Rodgers, US Department of Health and Human Services (HHS) consultant for technical writing support; and ICF librarians, abstractors, and HHS support staff. HHS and ICF staff collaborated with the Committee in the design and conduct of the searches by assisting with the development of the analytical frameworks, inclusion/exclusion criteria, and search terms for each primary question; using those parameters, ICF performed the literature searches. Sources of funding: E.A.H. has research support from the National Institute on Alcohol Abuse and Alcoholism (K01 AA028536-01).
Notes
As such evidence syntheses have proliferated, so too have the terms used to describe them, including narrative reviews, systematic reviews, meta-analyses, overviews of reviews, reviews of reviews, umbrella reviews, integrative reviews, meta-syntheses, and/or meta-reviews, among other terms.12 As Hennessy et al8 argued, the term meta-review is the most descriptive, and thus deserves to be used conventionally. Similarly, as we note in Table 1, a meta-analysis is a statistical approach but is often used to imply SR.
References
- 1.↑
Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc Sci Med. 2019;233:237–251. PubMed ID: 31233957 doi:10.1016/j.socscimed.2019.05.035
- 2.↑
Johnson BT. Toward a more transparent, rigorous, and generative psychology: editorial. Psych Bulletin. 2021;147(1):1–15. doi:10.1037/bul0000317
- 3.↑
Page MJ, McKenzie JE, Bossuyt PM, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol. 2021;134:103–112.
- 4.↑
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–1012. PubMed ID: 19631508 doi:10.1016/j.jclinepi.2009.06.005
- 5.↑
Torres A, Tennant B, Ribeiro-Lucas I, Vaux-Bjerke A, Piercy K, Bloodgood B. Umbrella and systematic review methodology to support the 2018 physical activity guidelines advisory committee. J Phys Act Health. 2018;15(11):805–810. PubMed ID: 30336718 doi:10.1123/jpah.2018-0372
- 6.↑
Johnson BT, Macdonald HV, Bruneau ML Jr, et al. Methodological quality of meta-analyses on the blood pressure response to exercise: a review. J Hypertens. 2014;32(4):706–723. PubMed ID: 24463936 doi:10.1097/HJH.0000000000000097
- 7.↑
Biondi-Zoccai G. Evidence Synthesis With Overviews of Reviews and Meta-Epidemiological Studies. Cham, Switzerland: Springer International; 2016.
- 8.↑
Hennessy EA, Johnson BT, Keenan C. Best practice guidelines and essential methodological steps to conduct rigorous and systematic meta-reviews. Appl Psychol Health Well-Being. 2019;11(3):353–381. PubMed ID: 31290288 doi:10.1111/aphw.12169
- 9.↑
Hennessy EA, Johnson BT. Examining overlap of included studies in meta-reviews: guidance for using the corrected covered area index. Res Synth Methods. 2020;11(1):134–145. PubMed ID: 31823513 doi:10.1002/jrsm.1390
- 10.↑
Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Soc Sci Med. 2019;233:237–251.
- 11.↑
Walsh D, Downe S. Meta‐synthesis method for qualitative research: a literature review. J Advan Nurs. 2005;50(2):204–211. doi:10.1111/j.1365-2648.2005.03380.x
- 12.↑
Ferrari R. Writing narrative style literature reviews. Med Writing. 2015;24(4):230–235. doi:10.1179/2047480615Z.000000000329
- 13.↑
Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Metho. 2014;5(4):371–385. doi:10.1002/jrsm.1123
- 14.↑
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to Meta-Analysis. Hoboken, NJ: John Wiley & Sons; 2011.
- 16.↑
2018 Physical Activity Guidelines Advisory Committee. 2018 Physical Activity Guidelines Advisory Committee Scientific Report. Washington, DC: U.S. Department of Health and Human Services; 2018.
- 17.↑
US Department of Health and Human Services. Physical Activity Guidelines for Americans. 2nd ed. Washington, DC: U.S. Department of Health and Human Services; 2018.
- 18.↑
Powell KE, King AC, Buchner DM, et al. The Scientific Foundation for the Physical Activity Guidelines for Americans, 2nd Edition. J Phys Act Health. 2018:16(1);1–11. doi:10.1123/jpah.2018-0618
- 19.↑
US Department of Agriculture. 2015 dietary guidelines advisory committee (DGAC) nutrition evidence library methodology. 2017. https://www.cnpp.usda.gov/sites/default/files/usda_nutrition_evidence_flbrary/2015DGAC-SR-Methods.pdf. Accessed February 24, 2021.
- 20.↑
Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD; 2014. AHRQ Publication No. (14)-EHC063-EF. https://effectivehealthcare.ahrq.gov/topics/cer-methods-guide/overview. Accessed February 24, 2021.
- 21.↑
Higgins JP, Green S, eds. Cochrane handbook for systematic reviews of interventions. version 5.1.0. The Cochrane Collaboration. 2011. http://handbook-5-1.cochrane.org/. Accessed February 24, 2021.
- 22.↑
Institute of Medicine. Finding what works in health care: standards for systematic reviews. 2011. ∼http://www.nationalacademies.org/hmd/∼/media/Files/Report%20Files/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards%20for%20Systematic%20Review%202010%20Insert.pdf. Accessed February 24, 2021.
- 23.↑
Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC. 2007;7:10–7.
- 24.↑
Cooper H, Koenka AC. The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. Am Psychol. 2012;67(6):446–462. PubMed ID: 22352742 doi:10.1037/a0027119
- 25.↑
Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–560. PubMed ID: 12958120 doi:10.1136/bmj.327.7414.557
- 26.
Stanley TD, Carter EC, Doucouliagos H. What meta-analyses reveal about the replicability of psychological research. Psychol Bull. 2018;144(12):1325–1346. PubMed ID: 30321017 doi:10.1037/bul0000169
- 27.↑
Pescatello LS, Corso LML, MacDonald HV, et al. Small sample sizes confound understanding of cardiometabolic responses to exercise. Exerc Sport Sci Rev. 2017;45(3):173–180. PubMed ID: 28419004 doi:10.1249/JES.0000000000000115
- 28.↑
Sandieson RW, Kirkpatrick LC, Sandieson RM, Zimmerman W. Harnessing the power of education research databases with the pearl-harvesting methodological framework for information retrieval. J Special Educ. 2010;44(3):161–175. doi:10.1177/0022466909349144
- 29.↑
Goossen K, Hess S, Lunny C, Pieper D. Database combinations to retrieve systematic reviews in overviews of reviews: a methodological study. BMC Med Res Methodol. 2020;20(1):1–15. doi:10.1186/s12874-020-00983-3
- 30.↑
Katzmarzyk PT, Powell KE, Jakicic JM, et al. Sedentary behavior and health: update from the 2018 physical activity guidelines advisory committee. Med Sci Sports Exerc. 2019;51(6):1227–1241. PubMed ID: 31095080 doi:10.1249/MSS.0000000000001935
- 31.↑
Kraus VB, Sprow K, Powell KE, et al. Effects of physical activity in knee and hip osteoarthritis: a systematic umbrella review. Med Sci Sports Exerc. 2019;51(6):1324–1339. PubMed ID: 31095089 doi:10.1249/MSS.0000000000001944
- 32.↑
Pescatello LS, Buchner DM, Jakicic JM, et al. Physical activity to prevent and treat hypertension: a systematic review. Med Sci Sports Exerc. 2019;51(6):1314–1323. PubMed ID: 31095088 doi:10.1249/MSS.0000000000001943
- 33.↑
Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V. Overviews of reviews. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 6.2 (updated February 2021). Cochrane; 2021. http://www.training.cochrane.org/handbook
- 34.↑
Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):1–27. doi:10.1186/s13643-017-0617-1
- 35.↑
Cooper H. Research Synthesis and Meta-Analysis: A Step-By-Step Approach. Newbury Park, CA: Sage Publications; 2015.
- 36.↑
Chau JY, Grunseit AC, Chey T, et al. Daily sitting time and all-cause mortality: a meta-analysis. PLoS One. 2013;8(11):e80000. PubMed ID: 24236168 doi:10.1371/journal.pone.0080000
- 37.↑
Sun JW, Zhao LG, Yang Y, Ma X, Wang YY, Xiang YB. Association between television viewing time and all-cause mortality: a meta-analysis of cohort studies. Am J Epidemiol. 2015;182(11):908–916. PubMed ID: 26568572 doi:10.1093/aje/kwv164
- 38.↑
Wilder J. The law of initial value in neurology and psychiatry: facts and problems. J Nerv Ment Dis. 1956;125(1):73–86. doi:10.1097/00005053-195701000-00009
- 39.↑
Pescatello LS, Buchner DM, Jakicic JM, et al. Response. Med Sci Sports Exerc. 2020;52(4):1003–1004. PubMed ID: 32175979 doi:10.1249/MSS.0000000000002264
- 40.↑
Pieper D, Antoine S, Mathes T, Neugebauer EAM, Eikermann M. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol. 2014;67(4):368–375. PubMed ID: 24581293 doi:10.1016/j.jclinepi.2013.11.007
- 41.↑
Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev. 2018;6(1):1–27.
- 42.↑
Oliveira NL, Botton CE, De Nardi AT, Umpierre D. Methodological quality and reporting standards in systematic reviews and meta-analysis of physical activity studies: a report from the Strengthening the Evidence in Exercise Sciences Initiative (SESS Initiative). 2021. doi:10.31222/osf.lo/krucz
- 43.↑
DiPietro L, Buchner DM, Marquez DX, Pate RR, Pescatello LS, Whitt-Glover MC. New scientific basis for the 2018 U.S. physical activity guidelines. J Sport Health Sci. 2019;8(3):197–200. PubMed ID: 31193291 doi:10.1016/j.jshs.2019.03.007
- 44.
Piercy KL, Troiano RP, Ballard RM, et al. The physical activity guidelines for Americans. JAMA. 2018;320(19):2020–2028. PubMed ID: 30418471 doi:10.1001/jama.2018.14854
- 45.
Piercy KL, Troiano RP. Physical activity guidelines for Americans from the US department of health and human services. Circ Cardiovascu Qual Outcomes. 2018;11(11):e005263. doi:10.1161/CIRCOUTCOMES.118.005263
- 46.↑
Thompson PD, Eijsvogels TMH. New physical activity guidelines: a call to activity for clinicians and patients. JAMA. 2018;320(19):1983–1984. PubMed ID: 30418469 doi:10.1001/jama.2018.16070
- 47.↑
Higgins JP, Jackson D, Barrett JK, Lu G, Ades AE, White IR. Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods. 2012;3(2):98–110. PubMed ID: 26062084 doi:10.1002/jrsm.1044
- 48.
Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med. 2002;21(16):2313–2324. PubMed ID: 12210616 doi:10.1002/sim.1201
- 49.
Mavridis D, Giannatsi M, Cipriani A, Salanti G. A primer on network meta-analysis with emphasis on mental health. Evid-Based Ment Health. 2015;18(2):40–46. PubMed ID: 25908686 doi:10.1136/eb-2015-102088
- 50.
Veroniki AA, Vasiliadis HS, Higgins JP, Salanti G. Evaluation of inconsistency in networks of interventions. Int J Epidemiol. 2013;42(1):332–345. PubMed ID: 23508418 doi:10.1093/ije/dys222
- 51.
White IR. Network meta-analysis. Stata J. 2015;15(4):951–985. doi:10.1177/1536867X1501500403
- 52.↑
Pescatello LS, Wu Y, Gao S, Livingston J, Shephard BB, Chen M. Do the combined blood pressure effects of exercise and antihypertensive medications add up to the sum of their parts? BMJ. 2021;7:E000895. doi:10.1136/bmjsem-2020-000895
