From Strength of Recommendation to Clinical Implementation: Weighing and Considering the Evidence

in International Journal of Athletic Therapy and Training

If the inline PDF is not rendering correctly, you can download the PDF file here.

Read not to contradict and confute, nor to believe and take for granted . . . but to weigh and consider.” — Francis Bacon (1561–1626)

A highly promoted element of evidence-based practice (EBP) is the Strength of Recommendation (SOR). The SOR is usually a letter grade (A, B, or C) and is a composite mark based on the quantity, quality, nature, and consistency of the external evidence. These SOR grades are most conspicuous to readers who are consuming Critically Appraised Topics (CATs) and Systematic Reviews, as well as Position Statements and Consensus Statements. The Strength of Recommendation Taxonomy (SORT)1 was developed by a consortium of family medicine physicians, and was designed to help clinicians rate the available external evidence (research) when used for clinical decision-making. Patient-oriented evidence—outcomes that were selected because of their direct impact on patients—is considered more favorable when compared to disease-oriented evidence.2 (For a quick review on patient-oriented vs. disease-oriented evidence, see Table 1.) Additionally, studies that tend to limit the threats to internal validity (studies that have low risk of bias) are also considered more favorable than studies with more risk of bias.2 Ultimately, from the SOR perspective, evidence that is generated from patient-oriented outcomes, and has little risk of bias, is ranked higher. Finally, there are elements of quantity and consistency that have to be incorporated into the SOR grade. The SOR grade is not created in the vacuum of a single study. The more studies that are included and the more studies included that “say the same thing” will also enhance the SOR grade. Briefly, the SOR is generated through a synthesis of the (1) quantity of external evidence used to answer a clinical question, (2) quality of the external evidence, (3) nature of the outcomes of interest, and (4) consistency of the results.

Table 1

Review of the Nature of Clinical Outcomes of Interest

Type of EvidenceDefinitionExample Outcomes of Interest
Patient-orientedEvidence derived from outcomes that focus on “what matters most to the patient”.Decreased injury, injury risk reduction following an injury prevention program, symptom improvements, enhanced quality of life, return-to-play, etc.
Disease-orientedEvidence derived from outcomes that a clinician can measure, but which may not necessarily be that important to the patient.Increased range of motion, increased muscle strength, limb girth, increased EMG activity of a muscle, reduction of inflammatory markers, etc.

Before we go into what the SOR means, we will quickly outline these four SOR inputs (see Figure 1):

  1. 1.The quantity of the external evidence:
    • One of the hallmarks of scientific research is the reproducibility of results, especially through independent investigations. In general, the SOR is based on more than one research study. In other words, “all” of the studies that address a particular research question are taken into account to generate the SOR grade. Multiple investigations on independent samples that all come from the same population of interest help to paint a clearer picture of the actual behavior likely found in that population. With quantity of the external evidence, more is better. . . but, not the only determining factor.
  2. 2.The quality of the external evidence:
    • As we have outlined in past editorials and papers,3,4 study designs can be ranked on a hierarchy that is based on potential bias that might be introduced by investigators. These biases can alter the interpretation of the internal and external validity of that study. There is an appreciable balancing act between control (internal validity) and generalizability (external validity).5 In general, studies with more control are higher on the hierarchy of clinical evidence. For example, cohort studies tend to have more control and therefore are ranked higher than case studies. In general, research studies that are higher on the hierarchy of clinical evidence have a better SOR grade.
  3. 3.The nature of the outcomes of interest:
    • In regard to the SOR, the key delineation for the nature of the outcomes of interest tend to be between patient-oriented evidence and disease-oriented evidence. As described above, patient-oriented outcomes are those with which the patient is more concerned. In contrast, disease-oriented outcomes are those that the clinician is interested in (not that the clinician is not interested in the patient-oriented outcomes!), and are often used to assist in determining the clinical course of action. Disease-oriented outcomes do not necessarily equate with the improvement in patient-oriented outcomes, but are sometimes used as surrogate measures.
  4. 4.The consistency of the results:
    • A fundamental requirement of scientific evidence (but often underappreciated!) is the ability to reproduce the results of a previous study. Findings from one well-designed study that used patient-oriented outcomes may be compelling, but consistent findings across multiple well-designed studies that used the same patient-oriented outcomes send a much stronger message. In this context, the consistency of results can be evaluated both qualitatively and quantitatively. We can qualitatively assess results and make a determination if the outcomes are “similar enough” to make an assessment of consistency. When we quantitatively pool results together because they share similar characteristics, we are performing a meta-analysis. Results that are consistent across studies will tend to improve the SOR.
Figure 1
Figure 1

—The natural process of re-weighting the likelihood of an outcome based on the external and internal evidence.

Citation: International Journal of Athletic Therapy and Training 24, 1; 10.1123/ijatt.2018-0130

In general, consistent results for outcomes that are patient-oriented receive an “A” grade. Patient-oriented evidence that is either limited in quality, quantity, or consistency is considered grade “B” evidence. Grade “C” evidence is based on either disease-oriented evidence (e.g., clinician-oriented outcomes) or clinical expertise. With that, there is a great deal of additional information available on the SOR. BUT, then what . . . how do we translate that grade into clinical practice?

Within Systematic Reviews, Meta-Analyses, CATs, the NATA Position Statements, and many of the Consensus Statements, investigators make recommendations and provide a SOR grade. However, at times, mistakes are made as to what the SOR is “telling us”. Here, the SOR is not signifying that the recommendation must be followed! The SOR provides an indication of the quantity, quality, consistency, and nature of the external evidence under which the results were generated. This means that under similar circumstances, the reported results are the most likely outcome. When we, as clinicians, alter the conditions (change the treatment, the outcome of interest, how the outcome is measured, etc.), we change the likelihood of our results matching those in the recommendation. Please note that we purposely did not say that we “decrease” the likelihood! There is a chance that the likelihood of a given result could increase.

We provide an intervention study example: Imagine that several randomized controlled trials (RCTs) consistently state that injury prevention programs (IPPs) reduce the risk of lower extremity injury in female soccer athletes. The recommendation of incorporating IPPs to reduce lower extremity injury risk in female soccer players would receive the SOR grade of A—consistent, patient-oriented evidence generated from high-quality trials. Just on face value, a clinician could make the supposition that by implementing an IPP without altering any of these factors, the same results (a decrease in lower extremity injuries in female soccer athletes) would probably happen. (We will come back to this example.)

Let’s take a brief detour to talk about RCTs. Although these are considered the highest level of individual clinical research design,3,5 can we say with certainty that the RCT can be easily reproduced in a clinical situation? There is a great deal of control in an RCT, but that cannot always be reasonably replicated in everyday clinical practice. This means that the generalizability of the RCT might need to be called into question. In other words, we “love” the RCT because it represents the study design with the lowest risk of bias (great internal validity), but its external validity, especially related to an individual clinician’s patients (population validity or clinical environment [ecological validity]), might be limited. Right there, we have to question the usefulness of the RCT—great internal validity, with limited external validity—is that easily accessible to the clinician? Maybe, maybe not. We can presume that the results of the RCT probably are true (have good internal validity), but it is very likely that we will have to alter the implementation of the intervention. This alters the consistency, therefore changing the likelihood that our results will match those presented in the recommendation.

Back to the example . . . Let’s imagine that a clinician evaluates this evidence and, based on that clinician’s appraisal and expertise (internal evidence), decides that an IPP is a good idea for her team. BUT, her team is men’s lacrosse. Should she just throw out the external evidence related to the IPP just because not all characteristics exactly match the characteristics as presented in the SOR? Of course not, and it would be silly to suggest that!

This is a purposely extreme example. Throwing out a strong recommendation for lowering injury risk, just because we cannot reproduce all components, is ridiculous. This is a fairly common misconception when considering the SOR. Rather, we should consider how the likelihood of similar results compared to those reported in the recommendation might be altered (increased or decreased) based on our individual clinical circumstances (also called “initial conditions”). At times, we evaluate a recommendation, particularly those with a high SOR, and see that the recommendation is untenable in our given clinical situation. Rather than it being a time to panic, this is an opportunity to be creative with adjusting the recommendation to fit our initial conditions, and then to re-weigh the likely outcomes. This re-weighting occurs naturally. As clinicians, we re-weigh the probability of events occurring all the time,6 most typically without “math” and oftentimes based on our own internal evidence7,8 and the patient’s evidence.9 Again, the SOR is not telling us what we “must” do, but how likely the outcome will be given the same set of initial conditions.

Recommendations graded as A are fairly easy to interpret clinically. However, at times, B and C grades can appear to be a bit more difficult to incorporate into clinical practice. If we take the same approach—not using the SOR to tell us what to do, but only as the likelihood of an occurrence—then it is easier to adjust the recommendation, as necessary. With both SOR grades B and C, we should initially evaluate why the recommendation was downgraded. Was it the quantity and quality of the external evidence (lower-quality study designs, expert opinion, standard of practice, etc.), nature of the outcomes (patient- or disease-oriented), or consistency of the results (or some combination) that brought that grade down? After that evaluation, we can more easily determine what adjustments seem reasonable and feasible in our own decision-making. The lower the SOR, the more uncertain the outcome when implemented in clinical practice. The less consistent the results, the more ambiguous our ability to make a clear decision becomes. Again, we re-weigh the evidence and make an expert judgment as to what might happen when we implement parts (or all) of a recommendation. Again, see Figure 1.

  1. 1.The probability of the clinical encounter:
    • Clinical recommendations from position statements, consensus statements, and systematic reviews frequently receive A grades for recommendations. While it is important to keep abreast of the best available external evidence, we must also consider the chance that we would actually encounter a particular injury or illness on which a given recommendation is based. For example, there may be strong evidence to support the use of a particular intervention for individuals who have suffered a hemorrhagic stroke, but if this is not the population with which a clinician typically works, how meaningful is this recommendation to that particular clinician? An individualized approach to determining the importance of each recommendation is necessary for each clinician.
  2. 2.Patient and clinician factors:
    • With regard to SORs, one of the most important things to consider is whether an individual clinician has the appropriate training, experience, and internal evidence to implement the recommendation effectively. This is particularly important to consider for students and new professionals. There may be grade A evidence that a particular clinical test is excellent in ruling in a particular diagnosis, but if the clinician lacks the appropriate skill in performing the test, the SOR is relatively meaningless in that instance. Furthermore, the preferences and values of the patients must be considered when making clinical decisions.9 Despite grade A recommendations from the research, patients are still free to choose the best course for them.
  3. 3.Clinical feasibility:
    • As mentioned in the example, we must consider if it is clinically feasible to implement recommendations. Do we have the facilities, person-power, resources, and equipment needed as described in a given recommendation? What alterations might we have to make? If we do alter the course, we need to consider how this might affect the likelihood of getting similar results.
  4. 4.Uncertainty of results and the ambiguity of the clinical options:
    • Lastly, especially for SOR grades B and C, we need to consider the level of uncertainty of the results and ambiguity of possible clinical options when implementing recommendations. Again, a re-weighting of the recommendation and the possible results should be performed through the lens of our own internal evidence. We should ask, “Is there enough compelling evidence for us to consider implementing the recommendation?” In instances where there is uncertainty, it is more difficult to support our clinical decisions using the external evidence alone. Again, all of our clinical decisions should be weighed against external, internal, and patient evidence.

In conclusion, the SOR is a very important component of EBP. The recommendations we encounter in Systematic Reviews and Meta-Analyses, Positon and Consensus Statements, and CATs need to be filtered through our ability to clinically implement them. Recommendation grades just evaluate the characteristics of the evidence, not whether or not an individual clinician should implement them. As we use EBP to advance the profession, it is essential for us to incorporate all sources of evidence (external, internal, and patient). Recommendations and their respective SOR are only as good as our ability to clinically implement them. Ultimately, we must continually evaluate our body of knowledge. How are we positioned with regard to the quality, quantity, nature, and consistency of the evidence for the recommendations that we make?

References

  • 1.

    Ebell MH, Siwek J, Weiss BD, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract. 2004;17(1):59–67. PubMed ID: 15014055 doi:10.3122/jabfm.17.1.59

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2.

    Medina JM, McKeon PO, Hertel J. Rating the levels of evidence in sports-medicine research. Athl Ther Today. 2006;11(5):38–41. doi:10.1123/att.11.5.38

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 3.

    McKeon PO, Medina JM, Hertel J. Hierarchy of evidence-based clinical research in sports medicine. Athl Ther Today. 2006;11(4):42–45. doi:10.1123/att.11.4.42

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4.

    Medina McKeon JM, McKeon PO. Grading clinical trials methods in sports medicine research. Athl Ther Today. 2009;14(3):4–9. doi:10.1123/att.14.3.4

    • Search Google Scholar
    • Export Citation
  • 5.

    Medina McKeon JM, McKeon PO. A balancing act between control and generalizability. Int J Athl Ther Train. 2016;21(2):1–3. doi:10.1123/ijatt.2016-0010

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 6.

    McKeon PO, Medina McKeon JM. Outcomes assessment: demonstrating our predictive ability as a healthcare profession. Int J Athl Ther Train. 2016;21(4):1–4. doi:10.1123/ijatt.2016-0062

    • Search Google Scholar
    • Export Citation
  • 7.

    Medina McKeon JM, McKeon PO. Bias is not a 4-letter word. Int J Athl Ther Train. 2018;23(3):93–95. doi:10.1123/ijatt.2018-0043

  • 8.

    McKeon PO, Medina McKeon JM. Evidence-Based Practice or Practice-Based Evidence: What’s in a Name? Int J Athl Ther Train. 2015;20(4):1–4. doi:10.1123/ijatt.2015-0055

    • Search Google Scholar
    • Export Citation
  • 9.

    Medina McKeon JM, McKeon PO. Patient evidence (AKA, boy, patients complicate our clinical practice). Int J Athl Ther Train. 2017;22(6):1–4. doi:10.1123/ijatt.2017-0088

    • Crossref
    • Search Google Scholar
    • Export Citation

If the inline PDF is not rendering correctly, you can download the PDF file here.

Jennifer M. Medina McKeon and Patrick O. McKeon are the Co-Editors in Chief of the International Journal of Athletic Therapy & Training (IJATT). They are in the Department of Exercise and Sport Sciences Faculty within the School of Health Sciences and Human Performance at Ithaca College, Ithaca, NY.

Address author correspondence to Jennifer M. Medina McKeon at jmckeon@ithaca.edu.
  • View in gallery

    —The natural process of re-weighting the likelihood of an outcome based on the external and internal evidence.

  • 1.

    Ebell MH, Siwek J, Weiss BD, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract. 2004;17(1):59–67. PubMed ID: 15014055 doi:10.3122/jabfm.17.1.59

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2.

    Medina JM, McKeon PO, Hertel J. Rating the levels of evidence in sports-medicine research. Athl Ther Today. 2006;11(5):38–41. doi:10.1123/att.11.5.38

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 3.

    McKeon PO, Medina JM, Hertel J. Hierarchy of evidence-based clinical research in sports medicine. Athl Ther Today. 2006;11(4):42–45. doi:10.1123/att.11.4.42

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4.

    Medina McKeon JM, McKeon PO. Grading clinical trials methods in sports medicine research. Athl Ther Today. 2009;14(3):4–9. doi:10.1123/att.14.3.4

    • Search Google Scholar
    • Export Citation
  • 5.

    Medina McKeon JM, McKeon PO. A balancing act between control and generalizability. Int J Athl Ther Train. 2016;21(2):1–3. doi:10.1123/ijatt.2016-0010

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 6.

    McKeon PO, Medina McKeon JM. Outcomes assessment: demonstrating our predictive ability as a healthcare profession. Int J Athl Ther Train. 2016;21(4):1–4. doi:10.1123/ijatt.2016-0062

    • Search Google Scholar
    • Export Citation
  • 7.

    Medina McKeon JM, McKeon PO. Bias is not a 4-letter word. Int J Athl Ther Train. 2018;23(3):93–95. doi:10.1123/ijatt.2018-0043

  • 8.

    McKeon PO, Medina McKeon JM. Evidence-Based Practice or Practice-Based Evidence: What’s in a Name? Int J Athl Ther Train. 2015;20(4):1–4. doi:10.1123/ijatt.2015-0055

    • Search Google Scholar
    • Export Citation
  • 9.

    Medina McKeon JM, McKeon PO. Patient evidence (AKA, boy, patients complicate our clinical practice). Int J Athl Ther Train. 2017;22(6):1–4. doi:10.1123/ijatt.2017-0088

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 76 76 23
PDF Downloads 24 24 8