Although there is a growing awareness of evidence-based practice among physical therapists, implementation of evidence-based practice has proved difficult. This article discusses barriers to access and interpretation of evidence. Some solutions are offered, including facilitating the publication of all research, use of an optimum format for reporting research, maximizing the efficient use of electronic databases, improving physical therapists' skills in critical appraisal of published research, and fostering consumer access to evidence. These strategies and others discussed in the article might facilitate implementation of evidence-based physical therapy.
Evidence-based practice is defined as the “integration of the best research evidence with clinical expertise and patient values.”1(p1) Within the international physical therapy community there is growing acceptance of this approach, which we refer to as evidence-based physical therapy.
Evidence-based physical therapy has become feasible with the enormous increase, in recent years, in the volume and accessibility of high-quality clinical research. One measure of the growth of evidence is that the Physiotherapy Evidence Database (PEDro) (http://www.pedro.fhs.usyd.edu.au/) now contains 398 evidence-based clinical practice guidelines, 3,920 randomized controlled trials (RCTs), and 713 systematic reviews that are relevant to the practice of physical therapy, 47% of which have been published since 1998 (based on a search of PEDro conducted on February 11, 2004).
It may be that developments in clinical practice have not kept pace with the increase in availability of evidence and that physical therapist practice has not changed as much as it might. Several studies2–5 have examined the use of evidence by physical therapists in clinical decision making over the past decade. A review of these studies indicates that much practice is still not evidence-based.
There are many reasons why research evidence may not translate into evidence-based physical therapy. These reasons include current health policies, the complexity of physical therapist practice, incomplete access to the evidence, difficulty interpreting the evidence, organizational barriers, and ineffectual continuing education programs.6
This article focuses on those barriers to evidence-based physical therapy that relate to access and interpretation of evidence. For simplicity, we have chosen to consider access to and interpretation of evidence only on the effects of interventions, and we ignore issues relating to implementation of evidence about other sorts of clinical questions (such as questions about diagnostic tests, prognosis, and so on). We restrict discussion to RCTs and systematic reviews of RCTs because these study designs provide the best evidence on the effectiveness of physical therapy interventions.7 An RCT is a study in which patients are randomly assigned to groups that do or do not receive the intervention of interest. A systematic review is a summary of literature that uses explicit methods designed to minimize bias in the location, appraisal, or synthesis of evidence on a review topic.
Access to Results of RCTs and Systematic Reviews
In the ideal world, the evidence-based practitioner would have immediate access to the results of all RCTs and systematic reviews at the point of patient care. This, of course, is not the current situation. There are a number of barriers to making evidence available to physical therapists. Completed studies may not have been published, published studies may be difficult to identify or retrieve, and relevant studies may not be available in the language of the user. These issues are discussed below.
Inspection of the Cochrane Central Register of Controlled Trials reveals that many RCTs in physical therapy are completed but never published. This is problematic because the likelihood of publication is related to trial findings. Medical trials with “negative results” (nonsignificant results or small effect sizes) are less likely to be published8 or have a greater time lag to publication8–10 than studies with positive results. Three studies8,11,12 that reviewed publication outcomes of trials submitted to ethics committees have each shown that publication is more likely if effects are large and statistically significant. Egger and colleagues13 estimated that the odds of publication are 2.4 times greater if results are statistically significant. Preferential publication of studies with significant results is problematic because it means that readers of clinical trials see an unrepresentatively positive subset of trials. As a consequence, readers may be inclined to form unrealistically optimistic opinions of the effects of interventions. This is termed “publication bias.”
The causes of publication bias are not well understood. Rosenthal14 called publication bias the “file drawer problem.” This term suggests that researchers choose not to report negative studies. The implication is that the source of the problem is with the individual researcher. There is some evidence that researchers are less likely to submit negative studies for publication.15,16 However, the responsibility also may lie with editors and journal reviewers. Mahoney17 asked reviewers to examine manuscripts with the same experimental procedures but with positive, negative, or mixed results. Positive manuscripts were more likely to be recommended for publication with only minor revisions. Reviewers more frequently rejected or recommended major revisions to negative manuscripts. Those manuscripts with mixed results were rejected consistently.
In some countries, the usual practice is to publish only positive trials.18,19 Vickers and colleagues18 reviewed the published reports of 1,352 controlled clinical trials and noted that 75% of trials arising from English researchers were positive, whereas Chinese and Russian (or USSR) researchers never reported negative results. Investigators from non-English-language countries are more likely to publish reports of trials in English-language journals if the trial results are positive. Egger and colleagues20 located 40 pairs of trials, each pair published by one author, where one trial was published in English and the other trial was published in German. Sixty-three percent of English publications reported statistically significant results, whereas this was the case for only 35% of German publications.
We are unaware of any study that has directly investigated publication bias in the physical therapy literature, but there is no reason to suspect that the physical therapy literature is not also subject to publication bias. The consequences of publication bias are not trivial—in our opinion, physical therapy intervention informed by a biased subset of the literature will not be as effective as intervention informed by all of the evidence.
Publication bias could be eliminated if all RCTs (and other sorts of studies) were published promptly, irrespective of outcome. This will only occur when there is universal acceptance among physical therapist researchers, funding agencies, and journals of the importance of prompt publication of completed trials. It may be possible to reduce the extent of publication bias by mandating prospective registration of clinical trials on registers such as the US clinical trials register (http://www.clinicaltrials.gov) and the United Kingdom's meta-register of controlled trials (http://www.controlled-trials.com). (Incidentally, trials registers also alert researchers to ongoing trials and so avoid unnecessary duplication of effort). However, registration is not yet common practice,21 and few physical therapy trials are registered. For example, only 52 trials with exercise or other physical therapy interventions are registered on the US clinical trials register (based on a search conducted on April 13, 2003). Ethics committees and funding bodies could promote more complete registration of trials by linking ethics approval or release of funds to trial registrations.
Prospective registration and prompt publication of the full results of completed trials would potentially improve the quality of evidence used by physical therapists. Publication means that the results are available to clinicians and so can be used to inform clinical decision making. Publication also means that the results of the trial can be considered for inclusion in systematic reviews and evidence-based clinical practice guidelines. In cases where trials are not promptly published, a record on a trial registry may alert a reviewer to the trial's existence. The reviewer can then contact the researcher who conducted the trial to obtain a copy of the trial methods and results and, if the trial is suitable, include the results in the review. Without trial registration, the results of the trial may have been missed.
Many physical therapists rely on readily available databases of health care literature (eg, MEDLINE, CINAHL, EMBASE) to access results of relevant research. Of these, only the MEDLINE database is available free of charge to individuals, although many therapists will have free access to CINAHL and EMBASE through institutional subscriptions. These databases index many trials and reviews of importance to physical therapists, but coverage is far from complete. This is because (1) most databases have a start date well after the first physical therapy trial was conducted and (2) the databases do not provide universal coverage of all of the journals that publish physical therapy research.
The earliest trial in physical therapy that we have been able to find is Colebrook's 1929 trial of ultraviolet irradiation for prevention of colds, infectious diseases, and chilblains and for promoting progress in schoolwork.22 This may be one of the first randomized trials ever conducted. Early trials such as Colebrook's trial are not indexed in any of the major literature databases. Although many journals commenced earlier, MEDLINE indexes publications starting from 1966, EMBASE indexes publications starting from 1975 and CINAHL indexes publications starting from 1982. Consequently, all 3 databases miss some physical therapy research. It is difficult to estimate how many trials in physical therapy predate each database, but to obtain a rough estimate we searched PEDro on April 17, 2003, and found that MEDLINE missed 21 trials, EMBASE missed 81 trials, and CINAHL missed 278 trials, solely because these trials were published prior to the coverage of the databases.
Not all databases provide comprehensive coverage of important physical therapy journals. Three studies23–25 have attempted to identify key physical therapy journals, and, because of the different methods used in those studies, each has produced a slightly different set of key journals. Both the Chartered Society of Physiotherapy (CSP) (http://www.csp.org.uk/libraryandinformation/library/physiotherapycollections/corelist.cfm) in the United Kingdom and the World Confederation for Physical Therapy (WCPT) (http://www.wcpt.org/programmes/ebp/journals.html) include lists of journals relevant to physical therapy and evidence-based practice on their Web sites. Table 1 summarizes coverage by major databases of key physical therapy journals as identified by the methods used in each study and by the CSP and WCPT. The number of journals identified ranged from 14 to 45. There was some overlap, but each set had at least 2 unique journals. CINAHL indexes 89% of the journals identified by Bohannon23 and 78% of the journals identified by Wakiji.24 Other databases appear to have more complete coverage, but in some cases the journals are incompletely indexed and indexing may extend back only a few years.
An example26 illustrates the poor coverage of the physical therapy literature by CINAHL. The “Evidence in Practice” section in the March 2002 issue of Physical Therapy described a clinical problem related to lymphedema.27 A search of the CINAHL database retrieved one evidence-based clinical practice guideline and one systematic review, but we believe it missed 6 RCTs indexed on the PEDro database. We present this information not to argue that PEDro is superior to CINAHL (we are unaware of any direct comparison of the databases with respect to coverage of the physical therapy literature), but to point out that searches of CINAHL can miss high-level evidence on physical therapy intervention.
Physical therapists who want to locate most relevant clinical trials could increase the sensitivity of their searches by searching multiple databases, although this is time-consuming for the busy clinician. Commercially available software permits simultaneous searching of more than one database. However, even when all 3 of the main medical literature databases (MEDLINE, EMBASE, and CINAHL) are searched, key journals, particularly non-English-language journals, are missed. Physical therapy-specific databases such as PEDro and the American Physical Therapy Association's (APTA's) Hooked on Evidence (http://www.apta.org/hookedonevidence) do not limit entries to certain journals or time periods, so they potentially provide a partial solution to this problem of coverage.
Access to Electronic Databases
As noted earlier, the MEDLINE database is available free of charge to individuals via PubMed (http://www.ncbi.nlm.nih.gov/PubMed/), and many therapists have free access to CINAHL and EMBASE through institutional subscriptions. In addition, many countries now have mechanisms designed to enhance clinicians' access to information. For example, in the United Kingdom the National Electronic Library for Health (http://www.nelh.nhs.uk/) provides access to MEDLINE, CINAHL, The Cochrane Library, Clinical Evidence, and some databases of guidelines to National Health Service employees. Most states of Australia have similar resources (eg, Clinical Information Access Program in New South Wales [http://www.clininfo.health.nsw.gov.au]). Several countries now provide free access to The Cochrane Library for all residents using National Provision Licenses (http://www.update-software.com/cochrane/provisions.htm). The PEDro database is available free of charge at http://www.pedro.fhs.usyd.edu.au, and Hooked on Evidence is available to APTA members at http://www.apta.org/hookedonevidence.
Access to Full Text
Searching for and appraising evidence is much easier if the entire journal article (ie, full text) is available online. There is some evidence that articles that are available as full text on the Internet are more likely to be accessed, read, quoted, and probably used in making decisions about patient management than those that are not available as full text. This is referred to as Full Text On the Net bias (FUTON bias).28
At present, most physical therapy journals that provide full-text versions of articles online do so for recent editions only, with a small number doing so for free (Tab. 2). Where full text is available, publishers commonly restrict access to subscribers. There are a number of document delivery companies that provide access to full text from physical therapy journals (eg, see list on http://www.apta.org/Research/factsheet_tips/howtofindresearch_related, accessed April 16, 2003), but the cost of this access is likely to prevent physical therapists from using this service regularly. Full-text access is often available in hard copies from libraries, but many therapists find that visiting libraries is prohibitively time-consuming. Access may be a greater problem for those working in smaller organizations or in regional areas.
One solution is for professional associations to subscribe on behalf of their members to journals that provide access to full-text articles online. Members of the association could then access the full text of articles in core journals from the association's Web site. Our provisional budget estimates suggest that physical therapy associations may be able to provide full-text access to a range of core journals for just a few dollars per member per year. The Australian Physiotherapy Association will begin offering this service to its members in 2004.
Electronic publication potentially provides a mechanism for providing low-cost access to evidence in developing countries. The World Health Organization's “Access to Research” initiative (http://www.healthinternetwork.net/) has helped 112 developing countries gain low-cost access to 2,100 journals. The Open Society Institute also provides electronic journal access (http://www.eifl.net/) to over 2,000 institutions in 39 countries.
Many trials are published in languages other than English. For example, approximately 8% of all randomized trials listed in MEDLINE are in languages other than English. PEDro now contains 236 RCTs and 20 systematic reviews in languages other than English (based on a search of PEDro conducted on April 13, 2003), and there are probably many trials and reviews in languages other than English that are not indexed in PEDro. A recent survey of LILACS, a Latin-American database, located 70 randomized trials published in Portuguese and Spanish (de Freitas A, Herbert R, Latimer J, Ferreira P; unpublished research). Some trials are published in a local language and in English, but it is unclear how common this practice is and, as discussed earlier, it is likely to occur to a greater extent with more positive studies. Thus, therapists who restrict their reading to English-language articles may be unaware of relevant evidence. The situation is likely to be even more acute for physical therapists who read only a language other than English.
Translating reports of existing high-quality trials would solve the language problem but would be associated with substantial costs. Nonetheless, it would be far cheaper, and quicker, to translate non-English-language reports of existing high-quality trials into English than to fund replication of trials by English-speaking physical therapists. Research funding bodies or professional associations in some countries may consider this to be a cost-effective way of increasing the volume of available research.
Interpretation of the Results of RCTs and Systematic Reviews
Once a trial or review has been accessed, it must then be read and interpreted. This involves critical appraisal of the quality of the study design (internal validity) and the applicability of the study to the clinical situation (external validity), and consideration of whether the size of the treatment effect warrants a change in clinical practice.29,30 The results of this appraisal process must then be applied to particular patients' circumstances.
Assessing Internal Validity
There are a number of key features of RCTs that have been shown to affect the validity of results (eg, blinding of assessors or patients13,31; concealed method of subject allocation to groups13,31,32). Trials without these design features will tend to show a greater effect of intervention.13 Our recent survey of 2,297 physical therapy trials revealed that 16% reported using concealed allocation (ie, at enrollment, the investigator is unaware of the group to which the patient will be allocated), 5% reported using blinded outcome assessors (ie, assessors who are unaware of which intervention the patient has received), and 9% reported blinding of patients.33 The typical randomized trial in physical therapy is potentially seriously biased.
Many health care professionals lack the skills and knowledge needed to discriminate between trials of low and high quality or to correctly interpret trial findings. A survey of general medical practitioners revealed that most did not understand fundamental terms used in evidence-based practice, such as “absolute risk reduction” and “number needed to treat.”34 Jette and colleagues' recent survey of a random sample of 1,000 APTA members35 suggests that similar problems exist among physical therapists. Less than 20% of respondents to the survey reported that they understood completely the terms “relative risk,” “absolute risk,” “odds ratio,” and “meta-analysis,” and only 55% reported that they were confident in their critical appraisal skills.
Therapists who have difficulty assessing the quality of clinical trials may find the “quality scores” in the PEDro database to be a useful guide. All trials indexed in PEDro have been rated for internal validity and the completeness of statistical reporting using the PEDro scale, an 11-item quality scale. The PEDro scale has demonstrated reliability36 and is based on the “Delphi list” of trial characteristics thought to be related to trial “quality” by a group of clinical trial experts.33,37 Scores for individual items (eg, subject blinding) also are presented. Ratings are used to rank the search results, so users are directed toward trials that are more likely to be valid and that contain sufficient statistical information to be interpretable. PEDro scores, however, can provide only a rough guide to trial quality. They do not take into account all relevant aspects of the design, conduct, and analysis of clinical trials, and they do not necessarily weight scale items optimally. Nonetheless, they may provide some guidance for readers.
Secondary sources of information also may be useful for physical therapists who lack critical appraisal skills. The results of clinical trials are summarized in a number of publications, including the ACP Journal Club, Evidence-Based Medicine, the “Critically Appraised Papers” section in the Australian Journal of Physiotherapy, and the Cochrane Database of Systematic Reviews. The first 3 examples distill the key findings of high-quality RCTs (and other types of studies), usually in one page or less, so they potentially also provide a significant time-saving mechanism for busy physical therapists. Cochrane systematic reviews can generally be considered to provide an unbiased synthesis of the literature because they are performed using stringent guidelines (including the critical appraisal of included trials).
Despite the demonstrated importance of study design in the size of study effects, we do not suggest that the results of suboptimally designed trials always should automatically be discarded. An imperfect trial may still provide more certainty than no trial at all. As such, we have suggested previously38 that a sensible approach for readers of clinical trials is to base the decision on whether to use the results of a potentially biased study in clinical decision making on the quality of other information that pertains to the clinical question at hand. If no less-biased information exists, the trial may be considered. However, we also note that, in our opinion, “there will usually be little point in reading clinical trials that do not meet basic criteria (true randomisation, acceptable follow-up, and blinding where possible).”38(p205)
In some trials, key features conferring internal validity may have been incorporated into the trial design, but these features are not reported when the trial is published. The reporting of RCTs could be improved by a greater use of the CONSORT statement39 (http://www.consort-statement.org/). The CONSORT statement comprises a checklist of key components of a trial's design, conduct, and analysis and a diagram for reporting the flow of subjects through the trial. This ensures key components are not omitted when researchers report their methods and data, which can improve the quality of reports of RCTs. As of January 29, 2003, 68 journals insist on use of the CONSORT statement when submitting trial reports for publication, but the CONSORT statement is yet to be adopted as policy by most physical therapy journals. Of the 84 key physical therapy journals identified by Bohannon,23 Wakiji,24 and Maher and colleagues,25 14 have adopted the CONSORT statement. A similar statement, the QUOROM statement, has been developed to improve the reporting of systematic reviews of RCTs.40
Systematic reviews provide an efficient way to access evidence on therapy because a single review can summarize the results of many single RCTs. However, there are a number of key features of systematic reviews that have been shown to influence the results. The search strategies used to identify relevant RCTs,41–43 the method of assessing RCT quality,44,45 and the method used to pool individual RCT results46 can all dramatically affect the conclusions of systematic reviews. This point is well illustrated by Ferreira and colleagues'46 reanalysis of 6 Cochrane reviews that used a system for pooling levels of evidence. Pooling is where the results of several studies are combined and summarized quantitatively. The authors found that the conclusion of a review could change dramatically depending on what levels of evidence system was used for pooling. For example, the 4 pooling systems produced the following conclusions for back schools: strong evidence that back schools are effective, weak evidence that back schools are effective, limited evidence that back schools are effective, and no evidence that back schools are effective.
It may be difficult to find a trial in which the sample characteristics and the frequency and exact method of administration of intervention match one's own practice. Differences between the characteristics of a study and the physical therapist's own practice can lead the therapist to disregard the results of relevant research. This may be appropriate if the closest research involves patients with completely different pathology or physiology. Results of a trial also may not be applicable if the intended patient has comorbidities that contraindicate the therapy.1 However, where the differences are less major, well-designed trials should not be discarded in favor of clinical experience alone. A more sensible approach may be to use relevant research to predict the treatment effect and to adjust that prediction according to clinical experience.29,47 For example, a trial by Moyer-Mileur and colleagues48 showed that daily range-of-motion exercises in preterm infants with very low birth weights improved growth rates and measures of bone mineralization. Infants were enrolled in the study at a mean of 2.2 weeks after birth, and all were tolerating enteral feeding. We might anticipate slightly better outcomes than those reported by Moyer-Mileur and colleagues if the exercise regimen were commenced sooner after birth. Conversely, a smaller treatment effect might be anticipated in an infant who is not tolerating enteral feeding.
Another issue faced by some physical therapists is that there are currently few RCTs conducted in their area of practice (eg, pediatrics, occupational health), or they treat people with relatively rare conditions (eg, cerebellar degeneration) for which it is probably not feasible to conduct trials. This means that the physical therapist may have to rely on lower levels of evidence or generalize from trials using subjects with a different diagnosis who may have some impairments in common with target patients. For example, there are no trials that have evaluated the effect of strength training for people with chronic whiplash-associated disorder, but a recent trial49 in women with chronic nonspecific neck pain showed clinically worthwhile effects of strength training on pain and disability. It seems reasonable to assume that there may be similar effects in people with chronic whiplash-associated disorder. Thus, these data could be used to estimate the size of effect of a similar intervention in this patient group. The process of estimation may be assisted by the routine provision of detailed information by authors (ie, explicitly reporting the source of patients in a trial [eg, volunteers, referral]), inclusion and exclusion criteria, and characteristics of the study population.
The terms “effective” and “ineffective” are commonly used to categorize physical therapy interventions, typically based on statistical hypothesis testing. Although it is very attractive to classify effects of intervention as effective or ineffective, more useful information can be gleaned from knowing the actual size of the effect.29,30 For example, although it is useful to know that gait training and treadmill training with body weight support for people with stroke is “effective,” it is more useful to know that, on average, this therapy increases walking speed by 0.25 m·s−1 compared with gait training alone.50 Classifying the outcomes of trials as effective or ineffective also may lead to apparent contradictions between the results of different studies of similar interventions where the studies have different statistical power.
We suggested in the previous section that to best apply the results of RCTs and systematic reviews to individual patient care, physical therapists need to adjust the average effect sizes found in studies to estimate the likely effect on an individual. However, this process is quite difficult and relies on the physical therapist's clinical experience with that patient group as well as knowledge of prognostic factors. It is likely that with practice a physical therapist's ability to do this would improve, but this is yet to be evaluated by research studies.
When interpreting the results of research studies, it is important to distinguish between situations where there is “evidence of no effect” (ie, a number of well-designed, adequately powered studies showed that a particular intervention did not have a clinically important effect in a particular setting) and situations where there is insufficient evidence to judge intervention effectiveness (ie, there have not been enough well-designed, adequately powered studies to assess the effect of a particular intervention in a particular setting). In the first situation, the confidence interval is narrow and includes the point of nonsignificance, so we can be fairly confident that the intervention is ineffective. In the second situation, we do not know whether the intervention is effective, ineffective, or harmful, because the wide confidence interval includes all of these possibilities. Anecdotally, these 2 concepts seem to be commonly confused by clinicians and, particularly, by health care policy makers. The distinction between the 2 situations is much clearer in studies in which confidence intervals about estimates of treatment effect size are reported.29,30,51
Evidence-based physical therapy has the potential to empower health care consumers to have real input into decision making about their care. If this is to happen, consumers will need to be able to access results of research, and they will need to be assisted to interpret this research. Ideally, both access and assistance would be provided by someone other than the treating health care professional. Informed decision making by health care consumers is becoming increasingly possible via mechanisms such as the Informed Health Online (http://www.informedhealthonline.org/item.aspx). This initiative of the Cochrane Collaboration provides lay summaries of Cochrane systematic reviews, many of which are of relevance to consumers of physical therapists' services (92 summaries identified using the terms “physiotherapy” and “exercise” on November 19, 2003). The initiative by some countries to provide residents with universal access to the Cochrane Library (http://www.update-software.com/cochrane/provisions.htm) greatly enhances the potential for increased consumer knowledge. Health departments in several countries also have set up Web sites aimed at providing information to consumers (eg, http://www.nhsdirect.nhs.uk in the United Kingdom, http://www.healthinsite.gov.au in Australia). However, more progress needs to be made in this area before a substantial proportion of patients are sufficiently informed with high-quality evidence to have informed input into decision making. Patients also should have the opportunity to assist in the development of research questions and funding priorities.
There are problems with access to evidence and interpreting whatever evidence is retrieved. Many of these problems are not unique to physical therapy. Some recent initiatives have improved access to research by physical therapists. Nevertheless, many barriers to access and interpretation of evidence remain. We have suggested some strategies that could be adopted by researchers, editors, reviewers, managers, and clinicians to overcome these barriers. These strategies include facilitating the publication of all research, use of an optimum format for reporting research, maximizing the efficient use of electronic databases, improving physical therapists' skills in critical appraisal of published research, and fostering consumer access to evidence.
In our view, the current problems with accessing and interpreting evidence are not a sufficient argument to reject evidence-based practice. Alternate models of practice (eg, those that emphasize clinical experience alone or the opinions of experts) also face problems, and, in our view, these problems are far more serious. More importantly, we believe that the current problems with accessing and interpreting evidence can be solved by the approaches we describe in this article.
The authors are Directors of the Centre for Evidence-Based Physiotherapy.
All authors provided concept/idea/research design, writing, fund procurement, and consultation (including review of manuscript before submission). Dr Maher provided data collection and analysis, project management, facilities/equipment, and institutional liaisons.
This work was partially funded by the Centre for Evidence-Based Physiotherapy's financial supporters: Motor Accidents Authority of New South Wales, Australia; Physiotherapists Registration Board of New South Wales, Australia; NRMA Insurance, Australia; and New South Wales Department of Health, Australia.
- Physical Therapy