HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) All Versions of this Article: ptj.20060280v1 88/2/270    most recent Submit a response Alert me when this article is cited Alert me when Rapid Responses are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Deutscher, D. Articles by Gutvirtz, M. Search for Related Content PubMed PubMed Citation Articles by Deutscher, D. Articles by Gutvirtz, M. Related Collections Professional Issues Medical Records Systems Social Bookmarking                      What's this?

Implementing an Integrated Electronic Outcomes and Electronic Health Record Process to Create a Foundation for Clinical Practice Improvement

D Deutscher, PT, MSc, is Quality Assurance Coordinator, Physical Therapy Service, Maccabi Healthcare Services, Tel Aviv, Israel. Mr Deutscher's institution mailing address is: 27 Hamered St, Tel Aviv 68125, Israel
DL Hart, PT, PhD, is Director of Consulting and Research, Focus On Therapeutic Outcomes Inc, White Stone, Va
R Dickstein, PT, DSc, is Department Chair, Department of Physical Therapy, Faculty of Social Welfare and Health Sciences, University of Haifa, Mount Carmel, Haifa, Israel
SD Horn, PhD, is Senior Scientist, Institute for Clinical Outcomes Research, Salt Lake City, Utah
M Gutvirtz, PT, MHA, BPt, is Head Director, Physical Therapy Services, Maccabi Healthcare Services

Address all correspondence to Mr Deutscher at: Deutsch_d{at}mac.org.il

Submitted September 19, 2006; Accepted August 28, 2007

	Abstract

 
Background and Purpose: Improving clinical outcomes requires continuous measurement and interpretation in conjunction with treatment process and patient characteristics. The purposes of this study were: (1) to describe implementation and integration of electronic functional status outcomes into an electronic health record (EHR) for the promotion of clinical practice improvement processes and (2) to examine the effect of ongoing outcomes data collection in a large physical therapy service in relation to patient and clinic burden.

Subjects: Data were examined from 21,523 adult patients (mean age=50.6 years, SD=16.3, range=18–99; 58.9% women, 41.1% men) referred for physical therapist management of neuromusculoskeletal disorders.

Methods: Process and patient characteristic data were entered into the EHR. Outcomes data collected using computerized adaptive testing technology in 11 outpatient clinics were integrated into the EHR. The effect of data collection was assessed by measuring the participation rate, completion rate, and data entry time. Qualitative assessment of the implementation process was conducted.

Results: After 1 year, the average participation rate per clinic was 79.8% (range=52.7%–100%), the average completion rate per clinic was 45.1% (range=19.3%–64.7%), and the average data entry time per patient (minutes:seconds) was 03:37 (SD=02:19). Maximum estimate of average administrative time per patient was 9.6% of overall episode time. Barriers to and facilitators of the implementation process were identified.

Discussion and Conclusion: The results indicate that routine collection of outcome data is realistic in a large public physical therapy service and can be successfully integrated with EHR data to produce a valuable clinical practice improvement platform for service evaluation and outcomes research. Participation and completion rate goals of 90% and 65%, respectively, appear to be feasible.

	Introduction

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 
The primary purposes of the physical therapy profession, according to the Guide to Physical Therapist Practice (Guide),¹ include enhancing physical and functional abilities and restoring, maintaining, and promoting optimal physical function, wellness, fitness, and quality of life as it relates to movement and health. The World Confederation for Physical Therapy has a similar purpose, described in the Declarations of Principle and Position Statements,² which emphasizes the importance of the activities and participation component of the World Health Organization's International Classification of Functioning, Disability and Health (ICF).³ The conceptualization of functioning as described in the Guide appears to be consistent with the ICF's conceptualization of activities and participation.⁴ Therefore, functioning, as defined by the ability to perform activities,^5,6 is important when establishing treatment goals for patients who are receiving physical therapy intervention for neuromusculoskeletal disorders. However, in Israel, physical therapists do not routinely measure physical function during daily clinical practice.⁷

Neuromusculoskeletal disorders, with their associated functional deficits, are the most common cause for referrals to outpatient physical therapy services, constituting more than 80% of the overall referral rate,⁷ with high prevalence and associated costs among adults.^8–13 Although previous studies^10,14,15 have demonstrated the use of data sets to study outcomes in outpatient physical therapy settings, no large-scale, multiple-site outpatient physical therapy service, to our knowledge, has published a method designed to implement a routine electronic patient-centered outcomes data collection process. In a single clinic project,¹⁶ which expanded to a small-scale multi-institutional project,¹⁷ an electronic informational system in physical therapy was described. The study included goal achievement, as perceived by physical therapists, as an outcomes measure. Later, Shields et al¹⁸ demonstrated use of a physical therapy computerized medical record combined with numerous outcomes measurement instruments to assist in standardization of data and clinical research in an acute care setting. These studies and others^19–22 had objectives similar to those of the current study: they described the use of electronic data retrieved from medical records and outcomes measurement tools to study relationships among patient characteristics, treatment processes, and outcomes. Vreeman et al²³ identified studies reporting on barriers to and benefits of implementing electronic health records (EHRs) in physical therapy, but no previous studies were identified describing a large-scale service using a routine EHR system with electronic patient-centered outcomes data collection.

As a consequence of the profession's aim to promote functioning, assessing functional ability has become important for understanding the contribution of physical therapy intervention to a person's ability to be an active participant in society.^4,24 Outcome measures in health care studies in general, and particularly in physical therapy, have evolved over the years toward a patient-centered approach through the use of patient self-report questionnaires.^4,25–30 The patient-centered approach for outcomes measurement has been defined by the Institute of Medicine (IOM) as providing care that is respectful of and responsive to individual patient preferences, needs, and values and ensuring that patient values guide clinical decisions.³¹ The IOM recommends the patient-centered approach as 1 of 6 aims for future health care systems.³¹ Patient-centered outcomes, as quantified via patient self-report questionnaires, have been recommended for physical therapy research,²⁴ have become well accepted in research,³² and increasingly have been integrated into clinical practice.^4,33–35 Many patient self-reported outcome measures have been shown to yield data with good internal consistency,^4–6,36 test-retest reliability,^37–39 construct validity^4–6,26,36 and predictive validity,^26,40 and sensitivity to change.^26,32,37,40

Outcomes data provide 2 main benefits. First, data potentially can assist physical therapists in clinical decision making while managing patients in real time. Second, when aggregating outcomes data with patient and treatment process data generated from an EHR system, a comprehensive clinical database can be created.²³ The database has the potential to become the foundation for ongoing effectiveness studies designed to assess relationships among outcomes, treatment procedures, and patient characteristics. These studies can be designed to determine those treatment procedures that, when administered to defined groups of patients, yield optimal functional outcomes in the real world of everyday practice.

We concluded that an observational clinical practice improvement (CPI) study design, as described by Horn and Hopkins,^41,42 includes the most adequate characteristics for assessing whether the intervention works in the real world of everyday practice or for whom the intervention works best. A CPI study is an observational cohort study that requires the attainment of prospective and retrospective data while not disrupting the natural process of intervention. The CPI study design examines what actually happens in the care process and contains several distinct features, some of which are meant to compensate for the shortcomings commonly attributed to observational studies, particularly the ability to account for patient covariates.^43,44 However, the ability to collect the outcomes data on a regular basis in a busy, multi-site health care service was unknown. We anticipated 2 main barriers related to a routine outcomes data collection process: (1) time to collect data and (2) clinicians’ acceptance and use of the data to facilitate clinical reasoning.

Therefore, the purposes of this study were: (1) to describe the implementation of an electronic patient-centered functional outcomes data collection system and integration of these data into an existing EHR and (2) to assess the effect of routine electronic outcomes data collection in a large-scale public health care service in relation to patient and clinic burden. It is our long-term goal to use the integrated EHR with its electronic outcomes to facilitate future CPI research projects for the purpose of improving clinical outcomes.

	Materials and Method

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 
Design

We conducted a prospective observational cohort study to evaluate the implementation of an electronic outcomes measurement system into routine physical therapist practice. Data on patient characteristics, treatment procedures, and functional outcomes were collected continuously from all patients who were referred for physical therapy intervention and who met the inclusion criteria described below. Treatment processes were not altered. Additionally, information regarding barriers and facilitators related to the implementation process of the outcomes measurement system was collected from the participating clinicians and service managers.

Subjects

The study was conducted in the physical therapy service of Maccabi Healthcare Services, a public health maintenance organization responsible for the health care of approximately 1.7 million people in Israel. The study included all patients at least 18 years of age who were admitted during 2005 for physical therapy intervention, secondary to a neuromusculoskeletal diagnosis. Patients were Israeli born and new immigrants from all over the world, including people whose primary language was Hebrew, Russian, Arabic, or English.

A total of 21,523 adults (mean age=50.6 years, SD=16.3, range= 18–99; 58.9% women, 41.1% men) were admitted for physical therapy during 2005, the first year of implementation. Health information for all patients was entered into the EHR, and they were eligible to take the functional surveys. Some patients had several episodes of care, for a total of 23,999 episodes. Participation in the outcome measurement system is described in Figure 1.

View larger version (19K): [in this window] [in a new window]   Figure 1. Participation in the outcome measurement system.

EHR

The Maccabi EHR system, which is based on the Clicks medical information system^* application, was originally developed to replace paper-and-pencil medical documentation. The EHR was designed to standardize data collection related to all medical encounters for all Maccabi Healthcare Services members, including medical data regarding treatments, patient responses, clinical assessment, and so on. Data related to therapy were entered into the computerized medical file by the therapists for each visit, creating a paperless work environment.

Detailed documentation of the medical encounter was accomplished using standardized data screens. Each patient has one central medical file, which can be accessed from any Maccabi Healthcare Services facility throughout the country, containing data related to therapy and medical encounters. The system was configured to provide online patient characteristics and treatment process data in real time for all clinicians treating the member. Data entered into the central data file for each therapy visit included patient personal identification number, age, sex, visit dates and times, treatment duration, clinician and clinic identification, episode identification, care type, type of payer, referring doctor information and diagnosis, therapy classification, treatment used, and information on falls for the elderly population. Discharge data, including reason for discharge, goal achievement, attendance and exercise adherence, and capacity to return to previous activities, were entered into the central data file by the therapist on closure of the episode of care.

Outcomes Data Collection

All data were collected via computer. Computerized outcomes measurement systems allow the application of advanced mathematical models used for measuring human abilities known as computerized adaptive testing (CAT)^{5,6,25,36,46–54} based on item response theory (IRT).^55,56 Using CAT to collect outcomes data in routine clinical work is a relatively new concept,^{5,6,36,47,48,53,54} but small-scale^49–52 and large-scale²⁶ applications have been described recently. The primary benefits of CAT are efficient data collection (ie, reduced respondent burden), with little loss of measurement precision,^{5,6,26,36,48–52,54,57–60} and the ability to integrate data collection with an EHR in an automated manner.

We used 8 body part–specific CATs²⁹ to estimate patient-specific measures of functional status (FS): cervical,²⁶ shoulder,⁵ elbow,²⁶ wrist and hand,²⁶ lumbar,³⁶ hip,⁶ knee,⁶ and foot and ankle.⁶ Briefly, CATs are developed by calibrating unidimensional items into scales using IRT methods.⁶¹ Measurements of FS generated using IRT mathematics and applied using CAT methods²⁹ were transformed to a range from 0 to 100 (low to high functioning) on a linear metric.^5,6,26,36 Functional status was defined operationally as the patients’ perception of their ability to perform functional tasks described in the FS items^5,6,26,36 and represents the "activity" dimension of the ICF.³

The 8 CATs function independently (ie, a specific CAT is administered to a specific type of patient). For example, a patient with a lumbar spine impairment will receive the lumbar spine–specific CAT.³⁶ For a patient with more than one impairment, the therapist identifies the primary impairment for which the patient is seeking therapy and selects the CAT that matches the primary impairment. Items are worded to reflect the impairment for which the patient is receiving therapy. Each CAT was developed for the specific purpose of assessing FS in a specific type of patient. Therefore, the item bank for each CAT is different (ie, different items and item calibrations). However, each CAT functions the same way: each CAT uses a specific starting item, uses the same item selection routine, estimates FS with the same technique using similar 0 to 100 linear metrics, and uses the same stopping rules. Specifics of each CAT have been published^5,6,26,36 but are beyond the scope and purposes of this article.

We selected these CATs to assess the FS of our patients because the measurements of FS generated using the CATs have been shown to have good reliability, validity, and responsiveness as tested with similar patients receiving therapy for neuromusculoskeletal disorders. For example, discharge FS measurements estimated using the CATs have been shown to have good discriminant validity for condition severity, symptom acuity, age, and surgical history as well as predictive validity (predictive ratios=0.961–1.057).²⁶ Measurements of change in FS (discharge FS – intake FS) using the CATs have been shown to have good discriminant validity for symptom acuity, age, medication use, impairment, and type of payer.²⁶ Functional status measurements using the CATs have large levels of responsiveness (effect sizes) for patients with the following neuromusculoskeletal impairments: cervical=0.88, shoulder=1.05, elbow=1.00, wrist or hand=0.94, lumbar=1.05, hip=0.86, knee=1.06, and foot or ankle=0.97.²⁶ Internal consistency reliability coefficients (Cronbach alphas) for individual item banks range from .92 to .97.^5,6,36 The above statistics were generated using patients treated in outpatient therapy clinics in the United States. Preliminary unpublished known-group construct validity study results using the same CATs in the Maccabi Healthcare Services system in Israel have produced similar responsiveness and known-group construct validity results.

The software used to collect the functional outcomes data was Patient Inquiry (PI) (version 5.0),⁶² developed by Focus On Therapeutic Outcomes (FOTO), Inc. This was the only identified computerized FS outcomes system that could provide the advanced measurement properties described above (ie, collection of outcomes data, via a CAT process, that could be integrated easily into the Maccabi Healthcare Services EHR).

We customized the outcomes data collection software before integrating the data into the existing Maccabi Healthcare Services EHR. Customization did not affect the CAT process. The original English-language outcomes survey questionnaires were translated into Hebrew, Russian, and Arabic using a method previously described.⁶³ A detailed description on the translation process is presented in the Appendix. In order to minimize patient or clinician burden, an electronic import process of demographic data (personal identification number, name, age, and sex) was managed by using the patients’ personal Maccabi Healthcare Services magnetic card in a card reader device, initiating an automatic computerized transaction from the Maccabi Healthcare Services main computer to the outcomes measurement software and database.

Because outcomes data were collected using CATs, each clinic had at least one computer survey stand, including a touch screen connected to the Maccabi Healthcare Services network. The application of the outcomes data collection process required 3 main steps: (1) implementation process, (2) survey administration, and (3) electronic database preparation.

Implementation Process

The implementation of the outcomes measurement system into the daily clinical work was managed by a precise, step-by-step procedure. The preparations were divided into 2 separate stages.

In stage 1, the Maccabi Healthcare Services physical therapy service management team investigated whether a daily, routine outcomes data collection was feasible.⁷ Maccabi Healthcare Services conducted a preliminary study (1999–2000) to assess cost-effectiveness in outpatient therapy (11 clinics, 6 months, 3,042 patients) using a 5-item, paper-and-pencil–administered functional and health-related quality-of-life survey.⁶⁴ The management team concluded that a patient-centered functional outcomes collection system could and should be incorporated into daily clinical work. The management team further concluded that the outcome measures will facilitate assessment of one patient at a time in real time, promote immediate feedback to the clinician, be responsive to functional change over time, and be linear. It was recommended that the data collection process be electronic, thereby reducing the burden on patients and clinics compared with paper-and-pencil survey administrations. Finally, it was concluded that the outcome measures should be integrated into the existing EHR database system, which was designed to collect data on process and patient characteristics.

In stage 2, the management team identified the outcomes measurement system, configured and programmed the software, initiated clinical education, and developed a monthly reporting process. The PI outcomes software⁶² was identified as the only existing system that could accommodate the demands listed above. The PI software was configured to match patient outcomes data with patient characteristics and treatment process variables existing in the EHR in order to develop the integrated electronic database.

Programming and testing were finalized during 2004, followed by a presentation of the project at a national meeting. A project coordinator was appointed in each of the 6 districts, and educational meetings were held in each of the 11 clinics participating in the study. These meetings included an introduction to outcomes in therapy, relevance and importance of patient-centered outcomes, a brief introduction to IRT and CAT, and instructions for the use of the configured software for outcomes data collection. During these meetings, a discussion was conducted regarding possible challenges related to the incorporation of routine outcomes data collection during busy daily clinical work. Clinicians were asked to collect information regarding barriers and facilitators of ongoing use of the new software in relation to clinic and patient burden. Patient-specific examples were given to emphasize the clinical use of the CAT scores during the clinical reasoning process and clinical management, emphasizing the need to reassess a patient's FS during the episode of care so the outcomes data could influence clinical decisions regarding treatment process or timing of discharge. Maccabi Healthcare Services started the 12-month data collection in 2005. Use of the outcomes data collection system was not mandatory during this period. Monthly reports were prepared for each clinic so that feedback on the percentage of patients using the system could be provided.

Survey Administration

Surveys could be administered on the first visit (intake survey), on one or more visits during the episode of care (status survey), and on the last visit (discharge survey). Intake surveys were administered using 2 optional methods: (1) in large clinics, with administrative personnel usually available, patients were instructed to arrive at the clinic 10 minutes prior to their appointed time so that they could receive a short introduction to the outcomes process by one of the administrative staff members, and (2) in smaller clinics, with no administrative staff available, the introduction and patient assistance with the survey were provided by the treating clinician during the first minutes of the visit.

The patients were introduced by the therapist to the use of the survey information to help measure and assess their functional ability throughout the episode of care. The patients were helped with the first 5 computer screens used for the survey set-up process. The set-up process was used to choose the preferred language, import patients’ demographic data from the EHR, choose the correct episode when multiple episodes existed in the outcomes database for the specific patient, and select the body part to be treated. Patients were asked to answer status surveys at the discretion of the treating therapist. Clinicians were instructed to collect status surveys at least every 3 or 4 visits. Therapists were advised to collect status and discharge surveys at the beginning of the visit so that the results of the survey could be used to assist in the management of the patient. Following each survey, a patient-specific report was printed including patient demographic data, FS data, the functional questions selected by the CAT process, and the patient's responses to the questions. Clinicians were encouraged to use the reported data in their clinical reasoning and decision-making process. For example, if a patient's FS score did not improve after 3 or 4 visits, clinicians were instructed to consult with their colleagues regarding other possible treatment strategies.

Electronic Database Preparation

Although some authors⁶⁵ have described an EHR as a system of integrated data sets that may include FS outcomes and a variety of health information about a person, that was not the case at Maccabi Healthcare Services. We integrated 2 data sets to form an aggregated physical therapy database, including the functional outcomes database and existing general EHR database. Each data set (EHR and outcomes) was stored on a separate central server located in Maccabi Healthcare Services headquarters and merged at the end of each quarter to create the final data set.

The existing EHR had been used to collect data for each physical therapy visit prior to the current project. Once the functional outcomes software was integrated with the existing EHR, member data were automatically shared between the 2 sets of software, which eliminated double entry of data. The following data were added to the EHR using the functional outcomes software: survey date; timing of computer use; FS questions and patient responses; potential risk-adjustment variables, including body part treated, symptom acuity, use of medication, surgical history related to the treated disorder, and physical activity prior to the initiation of the treated disorder^26,32; language spoken; and the CAT-generated FS measurements.

Measurements

The measures used to address the study purposes were participation rate, completion rate, and burden (patient and clinic). Participation and completion rates of an ongoing outcomes data collection process were considered important measures for assessing possible patient selection bias and patient and clinician acceptance of the outcomes measurement system. Participation rate was operationally defined as the percentage of patients meeting the inclusion criteria who started the data collection process. We set a participation rate goal of 90%, which would represent high patient acceptance and should be related to low patient selection bias. Completion rate was operationally defined as the percentage of episodes with FS intake data that also had FS discharge data from the patients and administrative data from the therapists. Because the completion of a discharge survey by a patient was driven mainly by clinician direction, we hypothesized that this measure would give insight into clinician acceptance of the outcomes process.

Preliminary experience in the Maccabi Healthcare Services suggested that 20% to 25% of patients who start treatment do not complete treatment due to a variety of reasons such as poor attendance.⁷ This group of patients was referred to as the "dropout" group and was described as a separate value in the completion rate analysis. Therefore, achieving a completion rate higher than 75% would be historically unlikely. Furthermore, we anticipated that an additional 10% decrease in completion rate would result due to participating patients who do not complete the functional survey at discharge. This group of patients was described by an "incomplete" value in the completion rate analysis. Finally, we anticipated that successful clinician acceptance would result in an increase in completion rate of up to 65%, which is consistent with completion rates of 50% to 70% in the United States using the same data collection system.²⁶ It should be noted that, during the first year, clinicians were encouraged but not mandated to have their patients collect FS data. Therefore, achieving a 65% completion rate would describe a high level of clinician acceptance of the option of using the outcomes measurement system for measuring their patients’ FS at discharge.

Patient and clinic burden were quantified in time units (ie, minutes and seconds). Patient burden was measured by survey data entry time (ie, time between the first data point entered and the last data point entered by the patient). Clinic burden was measured by the overall survey entry time throughout the full episode of care. Overall survey entry time was the sum of all time periods required by the patient to enter data, regardless of when the data were entered throughout the treatment episode.

To assess patient burden, we measured survey entry time. We assumed that the measurement system could be applied during the routine clinical work only if patients were able to finish the survey within an average of 5 minutes, so patient burden would be minimized and a high level of patient participation would be facilitated. The total timing of the survey included the time it took the staff to set up the survey in addition to the time it took the patient to answer the FS part of the survey. As described above, the survey set-up process was accompanied by the help of staff members and was timed only for surveys administered using the software upgrade installed on June 1, 2005, approximately at midterm of the first year. The FS part of the survey was completed by the patient without any help from staff members, unless the patient needed help or clarifications. Although the clinic staff members were not involved in the survey process itself, we decided to consider the full survey time (set-up and FS entry) as the entire time-consuming period used to assess patient and clinic burden.

To assess clinic burden, we measured total survey entry time. We assumed that, for a busy clinical setting to successfully implement the outcomes process, the overall time that it took staff and patients to answer all of the surveys throughout the episode of care should not exceed, on average, 15% of the total patient treatment time. Our assumption was that a patient should complete at least 2 surveys per treatment episode (ie, intake and discharge), but we recommended that 3 or 4 surveys be completed throughout the episode of care to facilitate patient management and clinical decision making. The average total treatment time throughout the episode of care was estimated by multiplying the average number of visits per episode of care by the average visit length of time, which is 25 minutes in Maccabi Healthcare Services.

Qualitative Information

Information regarding barriers and facilitators related to the implementation process of the outcomes measurement system was solicited during a series of mandatory meetings with the participating clinicians, clinic managers, and district physical therapy managers. Meetings were directed either by the first author or by district physical therapy managers. Participants were asked to elaborate on barriers and facilitators related to the implementation process, which were summarized by written reports published among all participating clinics. Barriers identified as being related to clinicians’ lack of education regarding technical or conceptual issues on the outcomes measurement process were addressed during and after the meetings. At the end of the study period, a summary meeting was directed by the first author and included all clinic and district managers and at least one clinician from each participating clinic. A summary report was published among all participating clinics and used for the qualitative analyses of the study.

Data Analysis

All analyses were conducted using SPSS, version 14. Descriptive statistics were used for analyses, including percentages for participation and completion rates and means, standard deviations, medians, and ranges for continuous data.

	Results

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 
Implementation Process

The final integrated FS/EHR database included a data set of 5,039 episodes of care for 4,845 patients having both an intake and discharge FS data and EHR patient demographic and treatment process data. The descriptive results of this integrated data set are presented in Tables 1 and 2.

View larger version (40K): [in this window] [in a new window]   Figure 2. Participation rate (N=23,999 episodes of care). Participation rate per month is the percentage of episodes for patients meeting the inclusion criteria who started the data collection process.

View larger version (51K): [in this window] [in a new window]   Figure 3. Completion rate (N=15,494 episodes of care). Completion rate is the percentage of episodes with functional status intake data that also had functional status discharge data from patients and administrative data from therapists. Dropouts is the percentage of episodes for patients who dropped out from treatment without any notice and, therefore, could not complete a discharge survey. Incomplete is the percentage of episodes for patients who, for no known reason, did not complete a discharge survey.

The main barrier related to clinician adherence was the prevailing belief that patients’ self-reported functional outcomes differed from clinicians’ perceptions of patient improvement. Discussion concerning specific examples revealed that, in most cases, clinicians believed the differences were related to the discrepancy between treatment goals as perceived by the clinician versus the patient's perception of functional change. In addition, some clinicians believed measuring outcomes was not important or did not have the potential to influence patient management. Some clinicians did not agree that patient-centered outcomes were valid and believed that pain reduction rather than FS should be the primary treatment goal. Some clinicians had difficulties implementing a new system into their routine clinical work, and many clinicians continued to use the outcomes system in a technical manner without synthesizing the functional information into their clinical reasoning process. Some clinicians expressed concern that Maccabi Healthcare Services management would use their patients’ outcomes information inappropriately. The main barrier related to clinic district managers’ adherence was that, when they were not positively involved in the implementation process, participation and completion rates were low.

The main facilitators related to patient adherence were that most patients had no difficulties completing the computerized survey, few patients refused to complete the survey, and most patients enjoyed answering questions related to their functional limitations. The main facilitators related to clinician adherence were that most of them thought that patient self-reported functional data facilitated functional goal setting at intake and functional goal achievement throughout the treatment episode.

Clinicians also found that the measurements supported their perceptions of the patients’ FS. Most clinicians believed measuring FS during treatment helped focus treatment management on functional goals, facilitated discharge planning for patients who did not improve during therapy, and facilitated communication with the patients regarding the patients’ functional difficulties, particularly when language issues caused communication difficulties. Furthermore, most clinicians thought that measuring FS was helpful in understanding the difference between changes in pain and functional change. They also stated that the measurements could be used for clinical research and that the use of the outcomes system supported professionalism and scientific credibility for the Maccabi Healthcare Services organization. The main facilitator related to clinic and district manager adherence was that most managers believed the outcomes system would help to accomplish the organization's goal of shifting from administrative information management to outcomes-based management. The final conclusion reached by all clinic and district managers was that the implementation process should be continued in all 70 Maccabi Healthcare Services physical therapy clinics.

Impact

The survey entry time (set-up and functional survey time) ranged from 0:00:29 to 5:34:30 (hours:minutes:seconds). Entry time for 0.9% of surveys was more than 30 minutes, which appeared to be a result of surveys initiated by error and ended after a long period of time, so these surveys were excluded from the analyses. That left 18,233 (99.1%) surveys from which the average survey entry time was calculated as 03:37 minutes (SD=02:19, median=03:08, range=0:00:29–0:29:55). The mean set-up time was 28 seconds (SD=0:01:28), and the mean functional survey time was 03:09 minutes (SD=0:01:46).

Clinic time for data entry was calculated by multiplying the average time needed for each survey by 4 maximum recommended surveys per episode of care (3:37 x 4 = 14:28 minutes) and will be regarded as 15 minutes for simplicity. The average total treatment time throughout the episode of care was calculated by multiplying the average number of visits per episode of care (6.25) by the average visit length of time (25 minutes), or 156.25 minutes. Therefore, the maximum percentage of overall episode time that could be devoted to survey administration was 9.6% (15/156.25 minutes), which was below the 15% upper limit that we anticipated.

	Discussion

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 
Our study was initiated by managers of a public physical therapy service who were determined to promote and improve outcomes for the benefit of their patients. From a management perspective, we argued that service managers should improve their ability to manage clinicians by using routinely collected data on patient self-reported outcomes. As a first step toward the goal of improved outcomes, the main purpose of this study was to evaluate the feasibility of implementing an electronic outcomes measurement system and integrating the outcomes into an existing EHR in routine physical therapist practice, forming an electronic database that could enable comprehensive service evaluation. To our knowledge, this is the first study to describe a process in which an electronic patient-centered FS measurement system used during daily clinical practice was integrated with an existing EHR system in a large public outpatient physical therapy service in Israel or elsewhere. The results support the initial success of the implementation process, which created an integrated EHR/FS database, comprising FS outcomes with enough patient characteristics and treatment process data to initiate CPI research projects with reasonable patient and clinic burden.

During the first 12 months of the implementation process, the participation rate increased from 50.1% to 79.8% (Fig. 2). Our goal was a 90% participation rate. The rate of participation by the end of the study among all 11 participating clinics ranged from 52.7% to 100%, with 5 clinics reaching or exceeding the 90% goal. During the study, some of the participating clinicians emphasized the potential for patient selection bias, suggesting that patients’ age could pose a barrier related to patient adherence, causing the participation rate to decrease with age. Therefore, we analyzed the participation rate during the last month of the study by 3 age groups. We found that the participation rate tended to decrease with age, with 83.8%, 80.3%, and 72.7% participation rates, respectively, for the age groups of 18 to 45, 46 to 65, and 66+ years, supporting the clinicians’ observation. Nonetheless, a large majority of elderly patients were able to use a computerized adaptive outcomes data collection system, which supported previous studies,^26,49,52 but future research should be conducted to study the relationship between age and participation rate.

Other barriers that might have a negative influence on participation rate were most likely to be clinician or management related. For example, it was suggested that low participation rates were associated with managers who presented collection of outcomes data as a low priority for their clinic, given their other interests. When we investigated further, we found that the managers of 2 clinics with relatively low participation rates of 52.7% and 68.2% had not established collection of outcomes data as a priority. The managers concluded that they had to be positively and actively involved to ensure that clinics and clinicians would reach predetermined goals. Obviously, the design of this study was not sufficient to determine which barriers were most important for a successful participation in outcomes data collection, but our data provide insight into issues that should be further studied in relation to improved participation rates. However, as 5 of the 11 participating clinics reached or exceeded the 90% participation rate goal and 4 other clinics approximated 90%, our data support our contention that the 90% goal is attainable for a large outpatient multi-site physical therapy system in Israel.

In contrast to participation rate, a high completion rate (Fig. 3) seems to be more difficult to attain. Overall completion rate increased during the study period from 27.1% to 45.1%, with clinic-specific results at the end of the first year ranging from 19.3% to 64.7%. These results support our estimation of a maximum attainable completion rate of 65%. Only 4 of the 11 participating clinics achieved a completion rate of over 60%. Qualitative assessment suggests management priority barriers impeded achievement of high completion rates similar to the participation rate barriers. High completion rates might be related to managers who were actively involved in monitoring, persuading, and educating clinicians to collect and use FS measures.

Several other barriers were suggested from discussion with clinicians and managers, including: a lack of knowledge of and experience in using outcomes data during the clinical reasoning process, a disconnect between some of the therapists’ perceptions of the constructs that should be measured (ie, impairments, pain, and satisfaction) and the constructs measured by the outcomes system (ie, FS), and uncertainties concerning the management team's interpretation and use of the outcomes measured (ie, therapist ranking by patient self-report outcomes).³² During the study, we emphasized that therapists were not mandated to collect FS outcomes data, so their personal perspectives could have influenced their participation and completion rates as well. Well-designed qualitative studies are recommended to assess reasons for low completion rates. These results highlight the complexity of implementing an outcomes measurement system into daily practice. We contend that, for high levels of completion rates to be achieved, more emphasis should be directed toward the education of clinicians in the use of outcomes to manage their patients. Furthermore, linking outcomes data collection and provider payment might improve completion rates over time.⁶⁶

The results demonstrated that, on average, a survey required an entry time of 03:37 (minutes:seconds), supporting a conclusion of minimal impact of ongoing electronic outcomes data collection. This result, in combination with the high participation rates, supported our conclusion that, in regard to patient burden, an outcomes measurement system based on computerized adaptive testing is highly feasible. This establishes the efficiency of CAT methods^5,6,36,48,54 described in simulation studies and recent prospective applications of CATs in postacute rehabilitation.^{26,49–52,58}

Results related to burden induced by the use of the outcomes measurement system on the clinical setting (ie, clinic burden, which we assessed by the percentage of overall episode time that could be devoted to survey administration per patient) supported a conclusion that minimal burden was added to the treatment episode (ie, 9.6% of treatment time). We assumed that a maximum of 15% of overall episode time could be devoted to survey administration before outcomes data collection became inappropriate in a busy clinical setting. Therefore, the 9.6% result supports our conclusion that the outcomes measurement system used by the Maccabi Healthcare Services system can be applied in routine daily practice without having a negative time-consuming impact on the clinic's function. The result of an average survey entry time of 03:37 (minutes:seconds) demonstrates that, in order to reach the upper limit of 15% overall episode time devoted for survey administration, an average of 6.5 surveys per episode would be needed. In reality, for the patient sample who completed their discharge survey (N=5,039 episodes), only 2.4 surveys per episode were completed (SD=0.75, range=2–11). This finding suggests that the estimation of a maximum average of 4 surveys completed throughout an episode of care was realistic. Furthermore, a wide range of improvement in implementation process leading to an increase in number of surveys completed per episode is possible before reaching negative time-consuming impact on the clinic's function. These results encourage us to consider measuring additional constructs to enhance more comprehensive clinical reasoning processes (eg, fear avoidance, depression).

The integration of outcomes data with patients’ demographic and medical background data (patient characteristics) and with treatment process data provides a wealth of information, including the 3 cornerstones needed for comprehensive CPI analysis.^41–43 We believe that the data set generated during this study⁶⁷ can provide information for determining those treatment procedures that, when administered to defined groups of patients, yield optimal functional outcomes in the real world of everyday practice. However, many other variables could be added to the data set in each of the 3 areas for more robust analyses. We intend to add data on the patients’ medical background, including co-morbidities and chronic use of medications, so that the interaction of these factors can be taken into consideration when analyzing the relationships between independent variables and functional outcomes. Furthermore, data on therapy appointment duration should be considered as part of the full data set so that evidence concerning optimal appointment times can be studied. These additional data exist already in the Maccabi Healthcare Services database and need to be merged with the integrated physical therapy data set. Future CPI research projects are under way in Maccabi Healthcare Service based on current findings, including preparations of CPI methodology components as described by Gassaway et al⁴³ (eg, identification of individual components of the care process, creation of a common intervention vocabulary and dictionary, identification of key patient characteristics and risk factors).

Limitations

Qualitative analyses were conducted using unstructured interviews; they were not designed to produce a high-level qualitative study, such as the study by Resnik and Jensen.⁶⁸ Therefore, our qualitative results must be approached with caution. The results should be used to plan future qualitative studies designed to assess barriers to continuous outcomes assessment.

The operational definitions of the treatment procedures have not been shown to be reliable or valid. Although these definitions did not affect the current study, clinicians could have identified a specific treatment but applied it in a different manner than other clinicians. Studies are recommended to standardize and improve the reliability and validity of the operational definitions of treatment procedures so that the validity of future CPI studies is improved.

We collected data in 11 clinics in Israel using proprietary software that used specific CATs. Therefore, the generalizability of our findings needs to be tested. As more CATs are developed and more outcomes data are collected using CATs,^25,26,46,58 additional findings concerning the CAT development steps, operational characteristics, measure equivalency, reliability, validity, responsiveness, and application efficiency will evolve.

Our description of patient and clinic burden using the measure of survey entry time involved data from 5 months after the start of the study, therefore, might not reflect the initial phases of implementation.

	Conclusions

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 
We have described implementation of an electronic outcomes data collection system into a system of outpatient clinics in Israel and integration of the outcomes into an EHR. The outcomes were designed to be used routinely during outpatient physical therapist management. The integrated database was designed to be used in future CPI studies. The data establish that the outcomes system can be implemented successfully and that a comprehensive electronic database can be constructed in a large, busy, multi-site, public outpatient physical therapy service. High participation rates and acceptable completion rates were attained, supporting good clinician acceptance of the outcomes process. Skillful clinical use of outcomes and managerial policy regarding collection and use of outcomes are believed to be factors contributing to successful implementation, but these beliefs await future studies. Nominal time to collect outcomes data using CAT technology suggests minimal patient or clinic burden. The electronic database generated by integrating the electronic outcomes data into an existing EHR appears to be well suited for future CPI analyses because it contains patient characteristics, treatment processes, and outcomes data.

	Appendix.

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 

View larger version (24K): [in this window] [in a new window]   Appendix. Survey Translation Process aCrane PK, Hart DL, Gibbons LE, Cook KF. A 37-item shoulder functional status item pool had negligible differential item functioning. J Clin Epidemiol. 2006;59:478–484.

	Footnotes

 
All authors provided concept/idea/research design. Mr Deutscher, Dr Hart, Dr Dickstein, and Dr Horn provided writing. Mr Deutscher, Dr Hart, and Dr Horn provided data analysis. Mr Deutscher provided data collection and project management. Mr Gutvirtz provided fund procurement and facilities/equipment. Dr Dickstein and Mr Gutvirtz provided institutional liaisons. Dr Hart, Dr Dickstein, and Dr Horn provided consultation (including review of manuscript before submission).

The authors thank Mrs Ilana Ariel, Mr Jacob Sela, Dr Ehud Kokia, and Dr Hagai Kedem for their vision and strong support for the inclusion of routine outcomes measurements in daily clinical practice. A special acknowledgment is dedicated to all participating therapists and their managers.

This study was approved by the Institutional Review Board for the Protection of Human Subjects of Maccabi Healthcare Services–HMO.

This work was funded by Maccabi Healthcare Services–HMO.

Dr Hart is an employee of and investor in Focus On Therapeutic Outcomes Inc, the database management company that manages the data analyzed in the study.

^* Roshtov Software Ind, 11 Omarim St, Omer 84965, Israel.

Focus On Therapeutic Outcomes Inc, PO Box 11444, Knoxville, TN 37919.

SPSS Inc, 233 S Wacker Dr, Chicago, IL 60606.

	References

Top Abstract Introduction Materials and Method Results Discussion Conclusions Appendix. References

 

1. Guide to Physical Therapist Practice. 2nd ed. Phys Ther. 2001;81:9–746.[Web of Science][Medline]
2. Declarations of Principle and Position Statements. World Confederation for Physical Therapy. 1999. Available at: www.wcpt.org/common/docs/WCPTPolicies.pdf.
3. International Classification of Functioning, Disability and Health (ICF). Geneva, Switzerland: World Health Organization; 2001.
4. Guccione AA, Mielenz TJ, Devellis RF, et al. Development and testing of a self-report instrument to measure actions: Outpatient Physical Therapy Improvement in Movement Assessment Log (OPTIMAL). Phys Ther. 2005;85:515–530.[Abstract/Free Full Text]
5. Hart DL, Cook KF, Mioduski JE, et al. Simulated computerized adaptive test for patients with shoulder impairments was efficient and produced valid measures of function. J Clin Epidemiol. 2006;59:290–298.[CrossRef][Web of Science][Medline]
6. Hart DL, Mioduski JE, Stratford PW. Simulated computerized adaptive tests for measuring functional status were efficient with good discriminant validity in patients with hip, knee, or foot/ankle impairments. J Clin Epidemiol. 2005;58:629–638.[CrossRef][Web of Science][Medline]
7. Deutscher D. Quality Assurance in Physical Therapy [master's thesis]. Tel Aviv, Israel: Tel Aviv University; 2002.
8. Carter SK, Rizzo JA. Use of outpatient physical therapy services by people with musculoskeletal conditions. Phys Ther. 2007;87:497–512.[Abstract/Free Full Text]
9. Cunningham LS, Kelsey JL. Epidemiology of musculoskeletal impairments and associated disability. Am J Public Health. 1984;74:574–579.[Abstract/Free Full Text]
10. Freburger JK, Carey TS, Holmes GM. Effectiveness of physical therapy for the management of chronic spine disorders: a propensity score approach. Phys Ther. 2006;86:381–394.[Abstract/Free Full Text]
11. Jette AM, Smith K, Haley SM, Davis KD. Physical therapy episodes of care for patients with low back pain. Phys Ther. 1994;74:101–110; discussion 110–115.[Abstract/Free Full Text]
12. Kelsey JL, Hochberg MC. Epidemiology of chronic musculoskeletal disorders. Annu Rev Public Health. 1988;9:379–401.[CrossRef][Web of Science][Medline]
13. Kelsey JL, White AA III, Pastides H, Bisbee GE Jr. The impact of musculoskeletal disorders on the population of the United States. J Bone Joint Surg Am. 1979;61:959–964.[Abstract/Free Full Text]
14. Di Fabio RP, Boissonnault W. Physical therapy and health-related outcomes for patients with common orthopaedic diagnoses. J Orthop Sports Phys Ther. 1998;27:219–230.[Web of Science][Medline]
15. Zigenfus GC, Yin J, Giang GM, Fogarty WT. Effectiveness of early physical therapy in the treatment of acute low back musculoskeletal disorders. J Occup Environ Med. 2000;42:35–39.[CrossRef][Web of Science][Medline]
16. Savander GR, Stutz RN. Electronic data processing of physical therapy services. Phys Ther. 1973;53:1046–1054.[Medline]
17. Savander GR. Development of an outcome assessment and informational system for physical therapy: a multi-institutional project. Phys Ther. 1977;57:891–896.[Web of Science][Medline]
18. Shields RK, Leo KC, Miller B, et al. An acute care physical therapy clinical practice database for outcomes research. Phys Ther. 1994;74:463–470.[Abstract/Free Full Text]
19. Brown M, Gordon WA. Rehabilitation indicators: a complement to traditional approaches to patient assessment. Cent Nerv Syst Trauma. 1986;3:25–35.[Medline]
20. Kaur K, Forducey PG, Glueckauf RL. Prototype database for telerehabilitation. Telemed J E Health. 2004;10:213–222.[CrossRef][Web of Science][Medline]
21. Lehmann JF, Warren CG, Smith W, Larson J. Computerized data management as an aid to clinical decision making in rehabilitation medicine. Arch Phys Med Rehabil. 1984;65:260–262.[Web of Science][Medline]
22. Sulton LD, Hardisty B, Bisterfeldt J, Harvey RF. Computerized data bases: an integrated approach to monitoring quality of patient care. Arch Phys Med Rehabil. 1987;68:850–853.[Web of Science][Medline]
23. Vreeman DJ, Taggard SL, Rhine MD, Worrell TW. Evidence for electronic health record systems in physical therapy. Phys Ther. 2006;86:434–446; discussion 446–449.[Abstract/Free Full Text]
24. Jette AM. Using health-related quality of life measures in physical therapy outcomes research. Phys Ther. 1993;73:528–537.[Abstract/Free Full Text]
25. Ader D. Developing the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007;45(5 suppl 1):S1–S2.
26. Hart DL, Connolly J. Pay-for-Performance for Physical Therapy and Occupational Therapy: Medicare Part B Services (Grant 18-P-93066/9-01). Department of Health and Human Services/Centers for Medicare and Medicaid Services; 2006. Available at: http://www.cms.hhs.gov/.
27. Johnson M, Holthaus D, Harvell J, et al. Medicare Post-Acute Care: Quality Measurement Final Report. Denver, Colo: University of Colorado at Denver and Health Sciences Center; 2001.
28. Lohr KN. Health outcomes methodology symposium: summary and recommendations. Med Care. 2000;38(9 suppl):II194–II208.
29. Mills CN. Computer-Based Testing: Building the Foundation for Future Assessments. Mahwah, NJ: Lawrence Erlbaum Associates; 2002.
30. Sangha O, Stucki G. Patient-centered evaluation of the illness sequelae of musculoskeletal diseases: overview of important outcome instruments [in German]. Z Rheumatol. 1997;56:322–333.[CrossRef][Web of Science][Medline]
31. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
32. Resnik L, Hart DL. Using clinical outcomes to identify expert physical therapists. Phys Ther. 2003;83:990–1002.[Abstract/Free Full Text]
33. Myers AM, Holliday PJ, Harvey KA, Hutchinson KS. Functional performance measures: are they superior to self-assessments? J Gerontol. 1993;48:M196–M206.[Abstract/Free Full Text]
34. Tager I, Swanson A, Satariano W. Reliability of physical performance and self-reported functional measures in an older population. J Gerontol. 1998;53:M295–M300.[Web of Science]
35. Werneke MW, Hart DL. Categorizing patients with occupational low back pain by use of the Quebec Task Force Classification System versus pain pattern classification procedures: discriminant and predictive validity. Phys Ther. 2004;84:243–254.[Abstract/Free Full Text]
36. Hart DL, Mioduski JE, Werneke MW, Stratford PW. Simulated computerized adaptive test for patients with lumbar spine impairments was efficient and produced valid measures of function. J Clin Epidemiol. 2006;59:947–956.[CrossRef][Web of Science][Medline]
37. Binkley JM, Stratford PW, Lott SA, Riddle DL; for the North American Orthopaedic Rehabilitation Research Network. The Lower Extremity Functional Scale (LEFS): scale development, measurement properties, and clinical application. Phys Ther. 1999;79:371–383.[Abstract/Free Full Text]
38. Fritz JM, Irrgang JJ. A comparison of a modified Oswestry Low Back Pain Disability Questionnaire and the Quebec Back Pain Disability Scale. Phys Ther. 2001;81:776–788.[Abstract/Free Full Text]
39. Hart DL. Test-retest reliability of an abbreviated self-report overall health status measure. J Orthop Sports Phys Ther. 2003;33:734–744.[Web of Science][Medline]
40. Walsh DA, Kelly SJ, Johnson PS, et al. Performance problems of patients with chronic low-back pain and the measurement of patient-centered outcome. Spine. 2004;29:87–93.[CrossRef][Web of Science][Medline]
41. Horn SD. Clinical Practice Improvement Methodology: Implementation and Evaluation. New York, NY: Faulkner & Gray; 1997.
42. Horn SD, Hopkins DSP. Clinical Practice Improvement: A New Technology for Developing Cost-Effective Quality Care. New York, NY: Faulkner & Gray; 1994.
43. Gassaway J, Horn SD, DeJong G, et al. Applying the clinical practice improvement approach to stroke rehabilitation: methods used and baseline results. Arch Phys Med Rehabil. 2005;86(12 suppl 2):S16–S33.
44. Horn SD, DeJong G, Ryser DK, et al. Another look at observational studies in rehabilitation research: going beyond the holy grail of the randomized controlled trial. Arch Phys Med Rehabil. 2005;86(12 suppl 2):S8–S15.
45. Hart DL, Wright BD. Development of an index of physical functional health status in rehabilitation. Arch Phys Med Rehabil. 2002;83:655–665.[CrossRef][Web of Science][Medline]
46. Cella D, Yount S, Rothrock N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45(5 suppl 1):S3–S11.
47. Fliege H, Becker J, Walter OB, et al. Development of a computer-adaptive test for depression (D-CAT). Qual Life Res. 2005;14:2277–2291.[CrossRef][Web of Science][Medline]
48. Haley SM, Coster WJ, Andres PL, et al. Score comparability of short forms and computerized adaptive testing: simulation study with the activity measure for post-acute care. Arch Phys Med Rehabil. 2004;85:661–666.[CrossRef][Web of Science][Medline]
49. Haley SM, Fragala-Pinkham M, Ni P. Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme. Clin Rehabil. 2006;20:616–622.[Abstract/Free Full Text]
50. Haley SM, Ni P, Fragala-Pinkham MA, et al. A computer adaptive testing approach for assessing physical functioning in children and adolescents. Dev Med Child Neurol. 2005;47:113–120.[CrossRef][Web of Science][Medline]
51. Haley SM, Raczek AE, Coster WJ, et al. Assessing mobility in children using a computer adaptive testing version of the Pediatric Evaluation of Disability Inventory. Arch Phys Med Rehabil. 2005;86:932–939.[CrossRef][Web of Science][Medline]
52. Haley SM, Siebens H, Coster WJ, et al. Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation, I: activity outcomes. Arch Phys Med Rehabil. 2006;87:1033–1042.[CrossRef][Web of Science][Medline]
53. Ware JE Jr. Conceptualization and measurement of health-related quality of life: comments on an evolving field. Arch Phys Med Rehabil. 2003;84(4 suppl 2):S43–S51.
54. Ware JE Jr, Kosinski M, Bjorner JB, et al. Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Qual Life Res. 2003;12:935–952.[CrossRef][Web of Science][Medline]
55. Hambleton RK, Swaminathan H. Item Response Theory: Principles and Applications. Boston, Mass: Kluwar; 1985.
56. Hambleton RK, Swaminathan H, Rogers HJ. Fundamentals of Item Response Theory. Newbury Park, Calif: Sage Publications; 1991.
57. Jette AM, Haley SM. Contemporary measurement techniques for rehabilitation outcomes assessment. J Rehabil Med. 2005;37:339–345.[CrossRef][Web of Science][Medline]
58. Jette AM, Haley SM, Tao W, et al. Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Phys Ther. 2007;87:385–398.[Abstract/Free Full Text]
59. Sands WA, Waters BK, McBride JR, eds. Computerized Adaptive Testing. From Inquiry to Operation. Washington, DC: American Psychological Association; 1997.
60. Wainer H. Introduction and history. In: Wainer H, ed. Computerized Adaptive Testing: A Primer. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:1–21.
61. Van der Linden WJ, Hambleton RK. Item Response Theory: Brief History, Common Models, and Extensions. New York, NY: Springer; 1997.
62. Patient Inquiry [computer program]. Version 5. Knoxville, Tenn: Focus On Therapeutic Outcomes. Available at: http://www.fotoinc.com/demoinstructions.htm).
63. Lewin-Epstein N, Sagiv-Schifter T, Shabtai EL, Shmueli A. Validation of the 36-Item Short-Form Health Survey (Hebrew version) in the adult population of Israel. Med Care. 1998;36:1361–1370.[CrossRef][Web of Science][Medline]
64. Deyo RA, Battié M, Beurskens AJ, et al. Outcome measures for low back pain research: a proposal for standardized use. Spine. 1998;23:2003–2013.[CrossRef][Web of Science][Medline]
65. Henry SB, Morris JA, Holzemer WL. Using structured text and templates to capture health status outcomes in the electronic health record. Jt Comm J Qual Improv. 1997;23:667–677.[Medline]
66. Porter M, Teisberg E. Redefining Health Care. Creating Value-Based Competition on Results. Boston, Mass: Harvard Business School Press; 2006.
67. Swinkels IC, Ende CH, de Bakker D, et al. Clinical databases in physical therapy. Physiother Theory Pract. 2007;23:153–167.[CrossRef][Medline]
68. Resnik L, Jensen GM. Using clinical outcomes to explore the theory of expert practice in physical therapy. Phys Ther. 2003;83:1090–1106.[Abstract/Free Full Text]

CiteULike

Complore

Connotea

Del.icio.us

Digg

Reddit

Technorati

This article has been cited by other articles:

				Y.-C. Wang, D. L. Hart, P. W. Stratford, and J. E. Mioduski Clinical Interpretation of a Lower-Extremity Functional Scale-Derived Computerized Adaptive Test Physical Therapy, September 1, 2009; 89(9): 957 - 968. [Abstract] [Full Text] [PDF]

				D. L. Hart, M. W. Werneke, S. Z. George, J. W. Matheson, Y.-C. Wang, K. F. Cook, J. E. Mioduski, and S. W. Choi Screening for Elevated Levels of Fear-Avoidance Beliefs Regarding Work or Physical Activities in People Receiving Outpatient Therapy Physical Therapy, August 1, 2009; 89(8): 770 - 785. [Abstract] [Full Text] [PDF]

				L. Resnik and D. L Hart Author Response Physical Therapy, September 1, 2008; 88(9): 1009 - 1011. [Full Text] [PDF]

This Article Abstract Full Text (PDF) All Versions of this Article: ptj.20060280v1 88/2/270    most recent Submit a response Alert me when this article is cited Alert me when Rapid Responses are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Deutscher, D. Articles by Gutvirtz, M. Search for Related Content PubMed PubMed Citation Articles by Deutscher, D. Articles by Gutvirtz, M. Related Collections Professional Issues Medical Records Systems Social Bookmarking                      What's this?