|
|
||||||||
Research Reports |
L Resnik, PT, PhD, OCS, is Postdoctoral Fellow, Center for Gerontology and Health Care Research, Brown University, 2 Stimson Ave, Providence, RI 02912 (USA) (linda_resnik{at}brown.edu).
DL Hart, PT, PhD, is Director of Consulting and Research, Focus On Therapeutic Outcomes Inc, White Stone, Va. He is also an investor in the company
Address all correspondence to Dr Resnik
Submitted November 14, 2002;
Accepted June 27, 2003
| Abstract |
|---|
±SD) (19±17 versus 29±22). Mean treatment duration was different between groups (32±11 days for the expert group versus 31±8 days for the average group). Discussion and Conclusion. The results challenge assumptions that extensive clinical experience is necessary to achieve superior patient outcomes, and they provide information about the relationship between therapist characteristics and patient outcomes.
Key Words: Clinical competence Expert clinicians Low back pain Outcome measurement Physical therapist practice
| Introduction |
|---|
|
|
|---|
One limitation of current theoretical models of physical therapist expertise is they have been developed through research on therapists sampled solely on the basis of years of experience or reputation.1014 Although Rothstein15 defined expert therapists as those who achieve the best clinical outcomes and it is generally presumed that there is a relationship between a practitioner's level of expertise and patient outcomes, this relationship has not been examined.1,1012 Although reputation is a recognized facet of being acknowledged as an expert,16 no research has explored the relationship between reputation and measures of treatment outcome. Although the relationship between years of clinical experience and better patient outcomes is often hypothesized,11,14,17,18 our literature search yielded no research articles supporting this hypothesis.
To our knowledge, expert clinicians selected on the basis of patient outcomes data have not been previously studied, nor have the outcomes data of experts identified through experience or reputation been analyzed. Past researchers did not evaluate whether patients managed by experts obtained better outcomes than patients managed by nonexperts, although this type of analysis was recommended.10,11 One impediment to studies on clinical outcomes of physical therapy experts has been the lack of credible and widely accepted operational definitions of improvement following intervention19 and the lack of a "gold standard" for patient self-report of health status20 to assess the outcome of intervention.
Over the past decade there have been advances in the measurement of treatment outcomes using health-related quality-of-life (HRQL) instruments. Health-related quality-of-life instruments allow assessment of patients' perceptions of activities they can do, how often they do them, and the level of functional difficulty they have performing them. These instruments quantify physical, psychological, and social dimensions reflecting HRQL. As a result, HRQL data have been recommended as outcomes measures for physical therapists21 and have been used to assess intervention outcomes in patients with a wide variety of health conditions.22 We believed that the use of patient HRQL outcomes would provide an alternative to identifying expert physical therapists on the basis of reputation or years of experience. Thus, if patients of some therapists reported much higher gains in HRQL compared with similar patients managed by other therapists, data would support our belief that the former therapists could be considered expert. The purposes of our study were to identify expert physical therapists by using patient self-report of HRQL and to compare the characteristics of therapists whose patients reported high levels of HRQL at discharge with the characteristics of therapists whose patients reported average levels of HRQL at discharge.
| Method |
|---|
|
|
|---|
Patients
The study sample was a convenience sample of 24,276 patients (Tab. 1) who received rehabilitation for low back pain syndromes in 1999 and 2000. These patients were managed by 930 therapists in 354 outpatient rehabilitation clinics in the United States (Tabs. 2 and 3). These patients were selected from a larger dataset of 57,917 patients with low back pain syndromes who were entered into the Focus On Therapeutic Outcomes Inc (FOTO) database23,* in the 24 months of 1999 and 2000.
|
|
|
|
Therapists
Nine hundred thirty physical therapists employed in 354 outpatient rehabilitation facilities in 39 states managed the patients. The majority of therapists held bachelor's degrees in physical therapy (57%). Fewer held master's degrees (36%) or doctor of physical therapy (DPT) degrees (1%) (Tab. 2). Most therapists (96%) did not report any type of advanced orthopedic certification, such as an Orthopaedic Certified Specialist (OCS), certification in manual therapy (MTC), or completion of an approved manual therapy residency program (AAOMPT).
Clinics
All interventions were performed at outpatient rehabilitation facilities, operationally defined as clinics where patients with neuromusculoskeletal impairments not requiring hospitalization or nursing services are managed (Tab. 3). Outpatient rehabilitation clinics commonly provide single-discipline or multidisciplinary services (eg, physical therapy or occupational therapy) and may employ a variety of licensed and unlicensed providers, including physical therapists, occupational therapists, nurses, athletic trainers, physical therapist assistants, occupational therapist assistants, or aides. The majority of clinics were hospital based (51%), with fewer being corporately owned (17%). Only a small percentage of clinics were identified as private practices, payer owned, or physicians' offices (Tab. 3).
Data Collection
FOTO data collection procedures have been described previously.19,25 Briefly, patients completed self-report health status surveys before their initial examination and following discharge from their rehabilitation episode of care. Patient demographic data were collected at intake. Clinicians entered number of visits and intervention dates at discharge. When a clinic began collecting FOTO data, clinicians provided information on their education and years of experience using a clinician registration survey, and staff provided information describing the clinic. Clinic staff were trained in the data-collection process and were instructed to survey all patients older than 14 years of age who could communicate in English24 at intake and discharge from rehabilitation. Data from patients and staff were entered on paper survey forms, which were submitted to FOTO, where data were checked manually for completeness. Data were entered into a computer database where computer programs checked again to ensure that data were complete and were within appropriate ranges for each variable. Data identified as incomplete or inappropriate were returned to the clinic for correction. Corrected data underwent the same data quality screening upon resubmission, improving database quality for analyses. Duration of the episode of care was calculated electronically from dates of intervention. Sixty-six percent of all patients in the FOTO dataset completed both intake and discharge surveys.
Outcome Measures
Three HRQL measures were utilized in this study: the OHS, the physical component summary (PCS) of 12-Item Short-Form Health Survey (SF-12) questionnaire,26 and the PF-10.24 These measures were calculated from 24 HRQL items, which have been described previously,19,27 that were asked at intake and discharge. Items included questions from the acute version (1 week recall) of the SF-36.22 The SF-36 has been studied in populations of patients with low back pain, and measurements of reliability and validity have been published.22,28 Other studies have demonstrated good reliability and validity statistics of the individual scale measures of the SF-3622,29,30 and the SF-12 summary measures.26,31
The OHS was chosen because it measures both mental and physical dimensions of health. Internal consistency of items in the OHS constructs with 2 or more items has been reported (
=.57.91).19,32 Internal consistency reliability statistics of the items of the OHS constructs19,32 are comparable to the internal consistency reliability statistics calculated from the same items embedded in the SF-3622 and the SF-12.19,32 Test-retest reliability of data obtained with the OHS was good (intraclass correlation coefficient [ICC(2,1)]=.92).27 Validity of data obtained with the OHS has not been examined, but there is evidence that an overall HRQL measure with similar items is responsive for patients receiving outpatient therapy.33
Overall health status scores are calculated by averaging scores from the 8 embedded HRQL constructs: general health (1 item from the SF-12),26 physical functioning (10 items from the PF-10),22 role physical (2 items from the SF-12),26 bodily pain (2 items from the SF-36),22 vitality (1 item from the SF-12),26 mental health (2 items from the SF-12),26 role emotional (2 items from the SF-12),26 and social functioning (1 item from the SF-12).26 The OHS physical functioning construct also includes 3 new questions pertinent to clients with upper-extremity impairments.34 Scoring of item responses followed published algorithms.22,34 Raw ordinal scores were transformed to interval scores varying from 0 to 100 for each question.22,34 Transformed item scores were grouped by construct and averaged to obtain the score for each of the 8 OHS functional scales.
The PCS26,31 is a summary measure representing the construct of physical functioning. Items included in the PCS were selected from the SF-36 subscales22 of physical functioning, bodily pain, role functioning, and general health. The PCS26 was originally designed to reduce respondent burden while maintaining measure precision of the original instrument, the SF-36.22 The PCS score is transformed to have a mean of 50 and standard deviation of 10 in the general population.31 Test-retest reliability and validity of data obtained with PCS measures are good.26,31 Test-retest reliability (r=.89) was assessed using previously collected data from repeat administrations of the SF-36, which was completed by people from the general US population as part of the National Survey of Functional Health Status.26 Test-retest reliability (ICC[2,1]=.82) for the PCS26,31 was assessed using prospectively collected data from repeat administrations of the 24 OHS items from patients receiving outpatient rehabilitation.27 Known group construct validity was assessed using previously collected data from the Medical Outcomes Study, an observational study of adult patients with chronic medical conditions.26 The PCS differentiated groups of patients on seriousness of physical condition, frequency of acute symptoms, and self-reported change in physical condition.26 The PCS was analyzed because scores from the PCS were sensitive to change over the physical therapy episode of care for patients with low back pain syndromes and PCS change scores differentiated groups of patients with low back pain syndromes who met their treatment goals compared with patients who did not meet their goals,35 and it discriminates change in HRQL for patients receiving outpatient acute work rehabilitation compared with patients receiving work conditioning or work hardening.19
The PF-10 is a subscale of the SF-36 that measures physical functioning.22 Studies support internal consistency22,30 and test-retest reliability22 as well as construct validity19,22,29 and content validity22,29 of data obtained with the PF-10 measure. Internal consistency of the PF-10 calculated from the FOTO instrument has been reported (
=.90 at intake and .91 at discharge).32 The PF-10 was analyzed because it is responsive to change for patients with spinal syndromes receiving outpatient therapy36,37 and discriminates change in HRQL for patients receiving outpatient therapy.32,38 The PF-10 scores vary from 0 to 100. For all scales, the higher the score, the higher the client perceives his or her physical functioning. Each HRQL measure represents an estimate of the current level of HRQL related to physical functioning (PF-10 and PCS) or health status (OHS).
Health-related quality-of-life measures have been recommended as reliable, valid, and responsive measures for patients with low back pain.36,3842 Research comparing the responsiveness of the SF-36 and condition-specific instruments for patients with low back pain has shown that data obtained with the PF-10 correlate well with data obtained with condition-specific instruments.38,43 Taylor et al28 reported that most scales of the SF-36 were able to detect change in patients who had improved after intervention and that the PF-10 and PCS were consistently responsive in a group of patients with low back pain and sciatica. Of the 8 SF-36 scales, Patrick et al38 reported that the physical functioning scale was the most responsive to change in a population of patients with sciatica.
Data Analysis
Risk adjustment.
Risk adjustment, also called "case-mix adjustment," is a statistical process used to control effects of confounding variables seen in patient populations.4446 Risk adjustment considers factors other than the health care intervention or processes of care that help explain variation in patient outcomes.44 Previous studies have demonstrated effects of certain patient characteristics on rehabilitation outcomes. For example, age,36 depression,36 comorbidity,36 acuity of symptoms,19 intake functional status,33,36 history of prior exercise,32 and history of surgery36 have been associated with change in functional outcomes in patients receiving rehabilitation.
Although control of variables that could affect dependent variables is necessary before comparing patient outcomes across groups of clinicians, no consensus exists describing the optimal risk-adjustment method.45,46 For this study, we used an approach to risk adjustment similar to the one used by Jette and Jette.36 First, univariate analyses were used to identify possible confounding variables among patient characteristics36 available in the 19992000 FOTO data. All variables shown in Table 1 were analyzed. Next, a general linear model (GLM) was developed for each of the 3 outcome measurementsthe OHS,19,27 PCS,26,31 and PF-10.22,24 Each patient characteristic found to be significant in univariate analyses was included in the models using a backward-deletion process. General linear models allow simultaneous control of continuous and categorical variables in the risk-adjustment process.36,47
The following independent variables were included in the GLMs: age, severity, sex, onset of condition, number of surgeries for condition, reimbursement, exercise history, and employment status. Age (in years) was entered as a continuous variable. Severity of the condition was a continuous variable represented by the intake OHS score. Onset of condition represents the number of days from onset of symptoms until beginning intervention (coded as 07 days, 814 days, 1521 days, 2290 days, 91 days to 6 months, and over 6 months). Number of surgeries represents number of surgeries for the low back (coded as none, 1, 2, 3, or greater than 3). Reimbursement was the primary source of the payment for the patient's physical therapy (coded as indemnity, litigation, Medicaid, Medicare, patient private pay, health maintenance organization [HMO] or preferred provider organization [PPO], workers' compensation, or other). Exercise history was a measurement of the patient's self-reported exercise frequency prior to the episode of physical therapy care. Exercise history was coded as at least 3 times a week, 1 to 2 times a week, or seldom/never. Employment status represents the patient's employment status at intake to physical therapy, including full-duty full-time work, modified work, employed but not working, previously employed and currently receiving disability, unemployed, retired, or student.
Responsiveness.
Effect sizes (ESs)48 and standardized response means (SRMs)48 of the 3 outcomes measures were calculated. All measures had moderate to large ESs49 and SRMs. Effect sizes were .83 for the OHS, .86 for the PCS, and .69 for the PF-10. Standardized response means were .87 for the OHS, .79 for the PCS, and .75 for the PF-10. Based on these calculations and the fact that the OHS, unlike the other measures, measures both mental and physical dimensions of health, we chose the OHS as the measurement for use in classifying therapists by their patients' outcomes.
Risk-adjusted outcomes.
Residual scores for each discharge outcome measure were calculated after general linear modeling and saved for use in the final group analyses. Residual scores are the difference between actual discharge scores and the predicted scores after risk-adjusted modeling. Units of residual scores are in original scale points. We then calculated mean residual patient discharge scores for the OHS for each therapist.
The aggregated scores were used to classify therapists by their patients' outcomes. We operationally defined an expert therapist as a physical therapist whose mean risk-adjusted patient outcomes were above the 90th percentile and an average therapist as one whose patient outcomes were between the 45th to 55th percentiles. Because existing literature does not, to our knowledge, include research on using clinical outcomes to profile therapists, our selection method was based on the assumption that therapists managing patients with outcomes in the top 10% would reasonably represent expert therapists, and that therapists managing patients with outcomes in the middle 10% would reasonably represent average clinicians.
Therapist comparisons.
Differences in mean number of visits per patient and mean intervention duration per patient were assessed for therapists in the average and expert groups using t tests. Differences between therapist years of experience, type of professional (entry-level) degree, record of advanced orthopedic certification, region of the country, and type of practice setting were assessed using chi-square tests for categorical measures and t tests for continuous measures.
| Results |
|---|
|
|
|---|
|
±SD) for therapist groups. For each outcome measure, patients of therapists classified as expert reported higher risk-adjusted discharge scores (OHS=78±6, PCS=47±3, PF-10=80±8) compared with patients of therapists classified as average (OHS=67±5, PCS=41±3, PF-10=68±8). Therapists classified as expert or average had similar numbers of years of clinical experience (8±8 years).
|
|
|
| Discussion |
|---|
|
|
|---|
Years of Experience
The assumption that expert therapists would have many years of clinical experience has guided sampling of therapists in prior studies of expertise.1012 Our finding that there was no difference in years of clinical experience between groups classified as expert or average challenges the assumption that experience is necessary to achieve superior patient outcomes.50 Although our finding was somewhat surprising, it confirms the findings of Constance,51 who found no effect of therapists' years of experience on patient outcomes.
Advanced Certification
Results of this retrospective analysis demonstrated clinical outcomes measured by patient self-report of health status were related to practitioners' advanced orthopedic certification. Therapists in the group classified as expert were more likely than therapists in the group classified as average to hold 1 of 3 types of advanced orthopedic clinical certificationOCS, AAOMPT, or MTC. However, we believe that our findings were influenced by our analytical methods (ie, the examination of advanced certification and patient outcomes for only 20% of the therapists, rather than the entire sample). Further analysis of data for the entire sample of therapists (shown in Tab. 8) revealed that almost 40% of therapists with advanced certification had outcomes at or below the 50th percentile. This finding suggests that there was no clear relationship between advanced certification and patient outcomes.
Our finding was also influenced by the small sample size of therapists participating with the FOTO database who held clinical certification in 1999 and 2000. Some therapists might have possessed the knowledge and skills of a certified specialist but had not received their certification. The clinician registration form used by FOTO allowed therapists to indicate if they were formally certified but afforded no method of tracking therapists who have continuing education but no formal certification. Although clinician credentials are periodically updated, the difference between formal certification and possession of clinical skills might have confounded the results. Further inquiry into the effects of therapist training and experience and factors contributing to therapist effectiveness is needed.
There is a paucity of research examining the effectiveness of practitioners with advanced training. Our review of the literature yielded only one study that analyzed clinical outcomes of therapists with OCS. Hart et al33 reported that patients of therapists with OCS had higher PF-10 scores as compared with patients of therapists without OCS. We found no research on clinical outcomes of residency graduates or graduates of miscellaneous manual therapy programs. More information is needed on the outcomes of practitioners with advanced orthopedic certification.
Risk Adjustment
Our large sample size allowed analysis of many levels of categorical variables. This may have improved our explanatory models, which explained more variance than previous models using the same commercial database.36 Intake scores were the greatest predictor of discharge scores, explaining 29% of the variance in OHS scores. The 2 other largest predictor variables in our model of OHS were onset of condition (explaining 4%) and reimbursement source (explaining 1.5%). Inclusion of additional predictor variables, such as socioeconomic status,36 fear and avoidance beliefs,52 and patient education level,36 might have enhanced the explanatory power of our study. Further research to identify characteristics that predict outcome for patients with low back pain and other common diagnostic categories is recommended.
Patients Per Therapist
Therapists from the group classified as expert had fewer patients in the database than therapists from the group classified as average. One explanation for this might be a smaller caseload for these expert therapists. A testable hypothesis is that therapists who manage fewer patients per day may spend more time with each patient, a factor that may be associated with better patient outcomes. We found no prior research to support or refute this hypothesis.
Limitations
Others have discussed the limitations of using retrospective data to answer questions that have not been determined a priori, including problems with missing observations, data control, and patient selection bias that might threaten external validity of commercial databases.36,53 Researchers cannot exercise control of data-collection quality with retrospective data. There is a possibility of clinician and patient error in completing items on HRQL forms in retrospective and prospective studies. Because data were retrospective, there was no method to monitor the training process of data collection or assess level of adherence to FOTO guidelines for data quality. FOTO checks data quality manually prior to entry and via computer once data are entered into the database. Because of this data auditing, entered data were complete and values were within expected ranges for each variable necessary for calculation of HRQL estimates. However, there was no process to monitor which patients were selected for data collection in the clinic and no method to assess the reason for loss to follow-up.
We included only patients who had completed intake and discharge forms in our analyses because we were studying patient outcomes. The percentage of completed FOTO forms (66%) surpassed previous reports of 64% (1996 database)37 and 28% (19931994 database).36 Differences between patients with complete forms and those with incomplete forms were evaluated, and potential effects of missing follow-up data on our analysis were considered. Because the rate of form completion at discharge was lower for patients of physical therapists in the group classified as average, differential rates of inclusion in data analysis may have biased results toward the null (ie, minimizing differences in patient outcomes between average and expert groups). However, direction of potential bias cannot be accurately predicted. Although some patients who did not complete an episode of care may have left physical therapy because of failure to improve or because their condition was worsening, others may have ended therapy early because they were feeling better. The latter supposition is supported by our data because patients failing to complete data collection were younger and had higher initial OHS and PF-10 scores. Patients without discharge surveys simply may have returned to their physician for a follow-up visit and not kept a subsequent physical therapy appointment. This problem illustrates clinical reality and the need for quality control and data auditing for a clinical database used for research.
Another limitation of utilizing previously collected data is the researcher has no control over the choice of data elements. We therefore had no method to obtain information that might have allowed classification of patients according to movement signs and symptoms or specific diagnosis thought to affect outcome.5458 Classification schemes improve homogeneity of samples for more powerful comparisons.54,55,59
There may be limitations in the exclusive use of intake and discharge HRQL measurements to measure benefits of treatment as these measures may not include all areas of importance to the clinician and patient.1 It is possible that aspects of physical therapy intervention, such as patient education, have lifelong health effects that cannot be captured with HRQL measurements or assessed at the time of discharge.1 In our study, there was no method to track long-term HRQL outcomes within the existing database. Although these measurements may not reflect the actual long-term effect of physical therapy intervention, other research has shown that discharge scores of the SF-36 are good indicators of long-term outcomes for patients with low back pain.60
There are limits to the generalizability of the conclusions from our study to a broader population of physical therapists or patients. We do not know if our sample is representative of physical therapy practices in the United States.
Our sample included only patients of physical therapists who participated in the FOTO database. No effort was made to contact clinicians who do not participate in this database. Because practices elect to participate with FOTO for a variety of reasons,19,36,61 there are threats to external validity of the findings.
Although we considered our selection criteria (90th percentile outcomes and above) to be a reasonable way to classify therapists as experts, we recognize other researchers might have chosen a different sample to represent expert therapists (such as the top quartile) or limited selection to a smaller group (such as the top 5%). Others might have used assessment of clinically important change concerning HRQL measures to select groups for comparison. Others might have used assessment of clinically important change concerning HRQL measures to select groups for comparison. There is debate in the literature62,63 as to what represents clinically important change in HRQL scores. Effects of different selection techniques await future analyses.
We classified and analyzed only 2 groups of therapists in this study: expert therapists and average therapists. We did not include a third group of below-average therapists. Selection of only 2 groups of therapists limited information available for analysis and minimized information about characteristics of therapists whose patients had below-average outcomes.
Because of these limitations, our findings should be considered as hypothesis generating rather than hypothesis testing.36 Our study provides new information concerning the relationship between therapist characteristics and patient outcomes and introduces a method for using outcomes data to gauge therapist expertise. Although an observational study of this nature does not provide the type of rigor afforded by a clinical trial, our design provides data-based hypotheses that can be tested in future clinical trials.
| Conclusion |
|---|
|
|
|---|
| Footnotes |
|---|
This study was approved by the Institutional Review Board for the Protection of Human Subjects, Nova Southeastern University, Fort Lauderdale, Fla.
* Focus On Therapeutic Outcomes Inc, PO Box 11444, Knoxville, TN 37939-1444. ![]()
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
Y.-C. Wang, D. L. Hart, P. W. Stratford, and J. E. Mioduski Clinical Interpretation of a Lower-Extremity Functional Scale-Derived Computerized Adaptive Test Physical Therapy, September 1, 2009; 89(9): 957 - 968. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. K Freburger and T. J Mielenz Invited Commentary Physical Therapy, September 1, 2008; 88(9): 1004 - 1006. [Full Text] [PDF] |
||||
![]() |
J. M Fritz Invited Commentary Physical Therapy, September 1, 2008; 88(9): 1007 - 1009. [Full Text] [PDF] |
||||
![]() |
L. Resnik and D. L Hart Author Response Physical Therapy, September 1, 2008; 88(9): 1009 - 1011. [Full Text] [PDF] |
||||
![]() |
L. Resnik, D. Liu, V. Mor, and D. L Hart Predictors of Physical Therapy Clinic Performance in the Treatment of Patients With Low Back Pain Syndromes Physical Therapy, September 1, 2008; 88(9): 989 - 1004. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. Resnik, D. Liu, D. L Hart, and V. Mor Benchmarking Physical Therapy Clinic Performance: Statistical Methods to Enhance Internal Validity When Using Observational Data Physical Therapy, September 1, 2008; 88(9): 1078 - 1087. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Deutscher, D. L Hart, R. Dickstein, S. D Horn, and M. Gutvirtz Implementing an Integrated Electronic Outcomes and Electronic Health Record Process to Create a Foundation for Clinical Practice Improvement Physical Therapy, February 1, 2008; 88(2): 270 - 285. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. V Jewell and D. L Riddle Interventions That Increase or Decrease the Likelihood of a Meaningful Improvement in Physical Health in Patients With Sciatica Physical Therapy, November 1, 2005; 85(11): 1139 - 1150. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |