| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Research Reports |
LM Fitzgerald, PT, MEd, PCS, is Assistant Professor and Director of Clinical Education, Department of Physical Therapy, School of Health and Rehabilitation Sciences, University of Pittsburgh, Pittsburgh, Pa 15261 (USA).
A Delitto, PT, PhD, FAPTA, is Professor and Chair, Department of Physical Therapy, School of Health and Rehabilitation Sciences, University of Pittsburgh.
JJ Irrgang, PT, PhD, ATC, Associate Professor and Director of Clinical Research, Department of Orthopaedic Surgery, University of Pittsburgh School of Medicine, Pittsburgh, Pa.
Address all correspondence to Ms Fitzgerald at: lfitzger{at}pitt.edu
Submitted February 22, 2007;
Accepted March 5, 2007
 | Abstract |
|---|
Subjects and Methods: The CIET was used to evaluate physical therapist student clinical performance from 1999 to 2003. Data from 228 student evaluations, a survey of 26 clinical instructors (CIs), and an item review by 7 faculty members were used to collect validity evidence. The relevance of items on the CIET was examined by the survey and the item review. Coefficient alpha was calculated to estimate internal consistency among the items. A Spearman correlation was used to examine the relationship between 2 measures of clinical competence. A repeated-measures analysis of variance (ANOVA) compared the student scores at each clinical time frame to confirm expected improvements in performance longitudinally. Evidence for practicality was collected by the CI survey.
Results: Based on the faculty item review and the CI survey, all items were representative of skills and behaviors considered important for a clinically competent physical therapist. The internal consistency (alpha) was .98 for the patient management items. The average correlation of the 2 measures of clinical competence was .76. The repeated-measures ANOVA was significant and demonstrated improved patient management scores as the student progressed through the program. The CI survey results indicated that 96% of respondents agreed or strongly agreed that the instrument was short and easy to use.
Discussion and Conclusion: The results of the study suggest that the CIET is representative of skills and behaviors necessary for students to perform at the level of a competent therapist and that the instrument is practical to use for busy clinicians. The CIET appears to be a valid tool for measuring student clinical performance and can be a time-efficient alternative for CIs in today's demanding clinical environment.
| Introduction |
|---|
 TopAbstract Introduction MethodResultsDiscussionConclusionAppendix.References |
|---|
For example, the Guide to Physical Therapist Practice3 includes a detailed description of the scope of physical therapist practice, preferred practice patterns, and tests and measures and interventions relevant to each practice pattern. In addition, the Guide for Professional Conduct4 is intended to serve physical therapists in interpreting APTA's Code of Ethics5 in matters of professional conduct and provides guidelines by which physical therapists may determine the propriety of their conduct. The Guide to Physical Therapist Practice, the Guide for Professional Conduct, and the Code of Ethics also are intended to guide the professional development of physical therapist students. Thus, it would seem logical to base a clinical performance evaluation tool at least in part on these documents. When considering the comprehensiveness of these documents, one of the major challenges is to balance the need for assessment in a broad range of practice dimensions while allowing for an instrument that is pragmatic and sensible.
Today's busy clinical environments require efficiency when evaluating clinical performance. The increasing demands on clinicians for productivity and documentation result in less willingness to serve as clinical instructors (CIs).6,7 In addition, other limitations within the clinical environment interfere with an accurate evaluation of a student's clinical performance. In a review of performance evaluation of medical students, residents, and practicing physicians, Printen and colleagues8 explored cognitive, social, and environmental factors that contribute unwanted sources of score variation (bias). They found that instructors have a 1- or 2-dimensional conception of clinical performance and typically do not recall details. Furthermore, favorable clinical performance is reported more quickly and fully than poor performance, often leading to overly generous performance evaluations. Printen and colleagues suggested that clinical performance evaluation systems should ensure broad, systematic sampling of clinical situations and require use of short instruments.
We set out to develop a clinical internship evaluation tool for assessing the performance of physical therapist students with the main purpose that it would evaluate the skills necessary for clinical competence and provide a short, easy-to-use form for our CIs. The intended uses for the instrument are to evaluate student progress, competence, and performance in the clinical environment; to determine specific areas for remediation; and to provide information for program evaluation. Our goal was to have an instrument that would evaluate a physical therapist student's performance relative to that of a competent clinician who can effectively and efficiently manage his or her patients and clients to achieve optimal clinical outcomes. Because our students would be in their final clinical setting for 1 full year, we also felt that the instrument should allow evaluation of student performance that will likely progress beyond the level of a competent clinician.
We did not feel that previously developed instruments for measuring clinical performance of physical therapist students met the needs of our program and CIs. The New York State Performance Evaluation Instrument and several evaluation tools developed by individual programs were competency based, but not based on present standards of practice such as the Guide to Physical Therapist Practice.9,10 The Blue MACS is a valid and reliable instrument with good acceptance by the clinicians who use it, but it evaluates individual skills rather then overall competencies.11,12
The most widely used instrument is APTA's Clinical Performance Instrument.13 The anchor point for this instrument is "at the level of an entry-level physical therapist," which our faculty members believed was inadequate in many ways. First, we sought to achieve a higher level of performance. Second, we believed that our CIs could more accurately judge our definition of entry-level performance (at the level of a competent, cost-effective physical therapist) than they could an "average entry-level graduate." Finally, we believed that, with curriculum changes that we knew would eventually end with the awarding of a clinical doctorate, we were committed to a graduate who would practice at the level of a competent clinician—that is, a clinician who can effectively and efficiently manage his or her patients and clients to achieve optimal clinical outcomes. Additionally, we felt that all of the instruments were too lengthy and time consuming for today's busy CI.
The purpose of this study was to describe the process of developing and providing evidence for validity of the Clinical Internship Evaluation Tool (CIET). This process included: (1) developing a conceptual framework and generating the items, (2) field testing the initial version for a semester and receiving feedback from CIs, (3) revising the instrument based on CI feedback, and (4) testing the final version of the instrument.
| Method |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionAppendix.References |
|---|
A committee of 3 faculty members who served as the clinical education team in our department developed the instrument. The Director of Clinical Education then implemented use of the instrument with our MPT and DPT professional-level students to evaluate their clinical performance. Data gathered from use of the CIET to evaluate clinical performance of the students from 1999 to 2003 were used to provide validity evidence for the CIET. In addition, other faculty in the Department of Physical Therapy completed an item review of the instrument, and a survey was developed and given to clinical faculty regarding use of the instrument. All of the data were used to demonstrate evidence related to content, internal structure, relationship with external variables, and practicality. The protocol was submitted to the University of Pittsburgh Institutional Review Board (IRB) and determined to be exempt from IRB review because the data were collected and used for educational purposes.
Development of the CIET
We began to develop the instrument by generating a list of all possible skills and behaviors that physical therapists should demonstrate using the APTA Guide to Physical Therapist Practice,3 A Normative Model of Physical Therapist Professional Education,16 our curriculum plan, and the Commission on Accreditation in Physical Therapy Education (CAPTE) criteria.17 From a review of this list, we determined that 2 main factors were important when assessing student performance, and we divided the instrument into 2 major sections: professional behaviors and patient management skills. The professional behavior section contained 3 items related to safety, 6 items related to professional ethics, 4 items related to initiative, and 5 items related to communication. The patient management skills section contained items related to examination (8 items), evaluation (3 items), diagnosis and prognosis (5 items), and intervention (8 items). The final version of the CIET had 42 items in total, with 18 professional behavior items and 24 patient management items.
A rating scale was selected for evaluating the students on each item. For the professional behavior section, we felt that it was most important to know the frequency of which a student was displaying the behavior. A 5-point rating scale was developed, which ranged from 0 ("never displays the behavior") to 4 ("always displays the behavior") (Appendix). To demonstrate acceptable professional behavior, the student had to achieve a score of 4 for every professional behavior item. Monitoring of student behavior was required if a student received a score of 3 ("most of the time displays the behavior") for any professional behavior item. Any score below 3 required remediation of professional behavior and could result in failure of the clinical internship if not corrected.
For the patient management section, the CI is asked to measure the student's performance against that of a "competent clinician," which was defined as a physical therapist who is "able to skillfully manage a patient in an efficient manner to achieve an effective outcome." Performance for each patient management item was rated on a 5-point scale from "well below" to "well above" a competent clinician (Appendix). Well below was defined as "Student requires a great deal of guidance including instructions and verbal cueing to complete a task." Below was defined as "Student requires some supervision or increased time to complete a task." At that level was defined as "Student is at the level of a competent clinician. Student can carry an appropriate caseload for your clinic and achieve an effective outcome for his or her patients."
Above was defined as "Student is performing above the level of a competent clinician in your clinic. Student's clinical skills are highly effective and demonstrate the most current evidence in practice. Student can carry a higher than expected caseload." Well Above was defined as "This is reserved for the master clinician or clinical specialist." We felt the CI should be able to grade the student above "at that level," anticipating that our students would be highly effective in the latter part of a 1-year-long clinical internship. In addition, we occasionally had students who were entering the field of physical therapy from another clinical discipline and could potentially achieve a "master clinician" level with our intensive clinical education program. The rating scale for the patient management section was assigned a numeric score from 1 ("well below") to 5 ("well above"), and the scores for the 24 items were summed. The total score was used in analysis of the patient management section for individual students, classes, and the validation of the CIET. For grading purposes, the expectation is that the students progress from midterm to final in the early clinical internships, although they do not have to achieve a score of 3 ("at that level"). During the 1-year-long clinical internship, the expectation is that they will achieve a score of 3 for each rotation.
Sample/Description of Program
The sample for evidence related to internal and external structure included all professional-level MPT and DPT students in the program who graduated from 2000 to 2006. Our MPT program was 2 years long, and the DPT program is 3 years long. Each year consists of 3 terms, as our program is year-round. Our MPT students completed 7 clinical internships during their program. The DPT students completed the same first 4 clinical internships, then a year-long clinical internship. Clinicals 1, 2, and 4 were part-time semester-long clinical internships. Clinical 3 was a full-time, 7-week internship completed by both our MPT and DPT students. These first 4 clinical internships were integrated with the didactic course work in years 1 and 2. The MPT students then went on to do 3 full-time, 7-week clinical internships (5, 6, and 7) at the end of their didactic course work. Our DPT students completed a year-long clinical internship in their third year. During the year-long clinical internship, the CIET was completed every quarter. Data for the MPT students' sixth clinical internship corresponded in time to quarter 1 of the year-long clinical internship, and data for clinical internship 7 corresponded to quarter 2 of the year-long clinical internship. Quarter 3 and 4 data apply only to students in the DPT program. The sample size was determined by the number of professional-level students who had a final evaluation completed during each clinical time frame (Tab. 1).
|
Evidence related to content of the CIET was collected from a sample of 7 faculty members from the Department of Physical Therapy who completed an item review of the instrument. None of these faculty members were involved in the initial development of items for the CIET. Evidence related to content and practicality was collected from clinical faculty who supervised DPT students during their final year-long clinical internship. A survey instrument was sent to 76 clinical faculty, of which 26 faculty members responded. These 26 faculty members were representative of the year-long clinical faculty: 46% from acute care sites, 38% from outpatient sites, 8% from pediatric sites, 4% from a rehabilitation site, and 4% from a women's health site.
Procedure and Methods of Analysis
After the CIET was initially developed, CIs attended a training session to learn how to use the tool to evaluate a physical therapist student's clinical performance. Thereafter, new CIs were trained either through periodic in-services at the clinical site or individually. The Director of Clinical Education carried out all training. Clinical instructors used the tool exclusively to evaluate the clinical performance of our physical therapist students. The CIs were asked to use the tool at the midpoint and end of all clinical affiliations, both part-time and full-time. Immediate feedback was provided to CIs if they were using the rating scale incorrectly. After using the CIET in the fall semester of the 1999 academic year, minor changes were made in wording and format, and then collection of validity evidence to support use of the CIET began.
To provide evidence to support our intended uses and interpretation of the CIET, we collected validity evidence related to content, internal and external structure, and practicality. In collecting evidence related to content of the CIET, we wanted to determine whether the items on the CIET were representative of clinically competent behavior for physical therapists. In addition, we wanted to know whether we were teaching these behaviors in our curriculum. We used the CAPTE criteria, the Guide to Physical Therapist Practice,3 and our program's objectives for and vision of entry-level education in developing the tool. This procedure ensured that all of the items were relevant and representative of current clinical practice and our program's vision.
An item review form was developed and completed by faculty members to provide further evidence for content of the CIET. Faculty members were asked to answer "yes" or "no" to 10 questions about each item on the CIET. Through the item review form, feedback was requested about the clarity of each item, the item's relevance to physical therapist practice, whether the item was representative of our curriculum, and whether the item could be biased. Each item review form was reviewed, and the information was summarized to determine whether any items were irrelevant or nonrepresentative or whether the item should be revised to improve clarity or prevent bias.
Finally, evidence related to content was obtained from a survey instrument developed and sent to CIs of students on the first year-long affiliation in 2003. A question was included to specifically determine whether the CIET allowed the CIs to adequately assess a student's clinical performance. The CIs were asked to indicate how much they agreed or disagreed with the statement by checking the appropriate box on a 4-point scale from "strongly disagree" to "strongly agree." In addition, there were 2 open-ended questions on the survey instrument: (1) "What items on the CIET do you believe are irrelevant to the student's performance?" and (2) "What items should be added to the CIET?" The frequency and percentage of each response category for each of the items on the faculty item review form and the CI survey were determined to evaluate the evidence related to content of the CIET.
The second type of validity evidence we were interested in collecting was related to the internal structure of the CIET. We wanted to know whether all of the items on the CIET contributed to the evaluation of clinical competency. We hypothesized that the items within the professional behavior and patient management domains would be highly related to other items within the same domain. As such, we hypothesized that each domain would be unidimensional. To evaluate this hypothesis, we performed a factor analysis of all items within each domain.
We used the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy to determine whether the data were appropriate for factor analysis.18 Sharma18 suggested that the KMO value should be at least .60, although KMO values greater than .80 are preferred. We used the scree plot and eigenvalue-greater-than-1 rule to determine the number of factors to extract.18 We evaluated the factor loadings to interpret the meaning of identified factors. For the one-factor model, we expected each item to have a factor loading of 
.50. If the factor analysis identified more than one factor underlying the item responses, we performed orthogonal and oblique rotations to clarify the factor structure. If the factor analysis indicated that the underlying item responses fit a one-factor model, we estimated the consistency of responses across items (ie, internal consistency) with coefficient alpha and determined the item-to-total scale score correlations. Coefficient alpha for each clinical time frame was calculated. Coefficient alpha also was calculated with each item sequentially deleted, and the differences between the values of coefficient alpha with and without the item were compared. If coefficient alpha substantially improved when a particular item was deleted, it might indicate that the item was not contributing consistent information to the measurement scale.
If the pattern of item responses was multidimensional, calculation of internal consistency across all items within the domain and the item to total scale score correlations would have been inappropriate and thus were not calculated. The factor analysis, coefficient alpha and item-to-total scale score correlations were determined separately for the professional behavior and patient management scales for each of the 9 clinical time periods to determine whether internal structure varied based on whether it was a part-time or full-time clinical or an early or later clinical.
In considering evidence to support the external structure of the CIET, we wanted to know whether the patient management section was measuring clinical competence as we had defined it. To answer this question, we added a global rating scale to the CIET. After scoring all of the items for a student, the CI was asked to respond to the question, "On a scale from 0 to 10, how does the student compare with a competent clinician who is able to skillfully manage patients in an efficient manner to achieve effective patient or client outcomes?" The CI responded by placing an "X" on a scale from 0 to 10, with 0 being "well below a competent clinician," 5 being "at the level of a competent clinician," and 10 being "well above a competent clinician."
We hypothesized that, if the patient management items were measuring clinical competence, there should be a high correlation between the patient management score and the score on the global rating scale of clinical performance of the student. We assessed this with Spearman correlation coefficients calculated separately for each of the 9 clinical time periods except clinical 1 because the patient management section is not completed for this clinical. We felt that the Spearman correlation coefficient was most appropriate for this analysis because the global rating of clinical performance of the students was ordinal and the total professional behavior score was quantitative.19 A scatter plot of the data for each clinical period was inspected to determine whether the relationship was linear, with no apparent outliers, and demonstrated homoscedasticity prior to performing the correlation.
We also hypothesized that, as the students progressed through their clinical education, there should be an increase in their patient management scores over time. This hypothesis was assessed with a repeated-measures analysis of variance (ANOVA) for clinical internships 2, 3, and 4 and all 4 quarters of the year-long clinical for those students completing the DPT curriculum. The patient management section of the evaluation is not completed for the first clinical. Tukey post hoc tests were conducted to evaluate the pair-wise comparisons between clinical time frames.
Finally, we were interested in whether we had developed a tool that was easy and practical for the CI to use. A question about the practicality of the CIET was included on the survey instrument sent in 2003 to the CIs: "Is the test short and easy to use for CIs, making less demands on their limited time for clinical training?"
| Results |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionAppendix.References |
|---|
The survey of CIs had a 35% response rate; 26 of 75 CIs returned the survey questionnaire. In response to the item, "The Clinical Internship Evaluation Tool allowed me to adequately assess your student's performance," 4 respondents (16%) disagreed, 20 respondents (80%) agreed, and 1 respondent (4%) strongly agreed. One clinician did not answer that question. In response to the open-ended question, "What items on the CIET do you believe are irrelevant to the student's performance?," no items were believed to be irrelevant. Comments were limited to wanting clearer definitions of some items. In response to the second open-ended question, "What items should be added to the instrument?," 80.8% felt that no new items should be added, whereas 19.2% suggested that some items be expanded. For example, one CI felt that the item on legal issues could be clarified.
Evidence Based on Internal Structure
For the factor analysis of the professional behavior section, the KMO values ranged from .583 to .715, which implies that the data were adequate for factor analysis.18 Three factors appeared to emerge based on the eigenvalue-greater-than-1 rule and the scree plots, but items did not consistently load on the 3 factors across the 9 clinical time points. Communication, initiative, and professional behavior items appeared to load as more distinct factors, whereas safety loaded on all 3 factors. Because the professional behavior scale did not conform to a one-factor model, we decided that it was not appropriate to combine all of the items into a single score; thus, we did not calculate coefficient alpha or the item to total score correlations for this scale.
For the factor analysis of the patient management section, the KMO value was greater than .95 for all clinical time frames, which is considered "marvelous."18 Only one distinct factor was extracted based on the eigenvalue-greater-than-one rule and scree plots for each clinical time period. The eigenvalues for the first factor ranged from 15.7 to 17.9, and the eigenvalues for the next factor were 1.5 or less. The eigenvalue for the second factor was only greater than 1 for 2 clinical time frames, and loadings on that component were all less than .50. Factor loadings for all items on factor 1 ranged from .743 to .883 (Tab. 2).
|
Evidence Based on Relationship With Other Variables
In examining the mean score for patient management for each clinical time frame, an increase was observed as the students progressed through the clinical internships (Tab. 3). The correlations of patient management scores to the global ratings of clinical performance of the students ranged from .54 to .89, with an average correlation of .76 (Tab. 4). The correlations for clinicals 3, 4, and 5 and quarter 4 were very strong (r>.80). Correlations for clinical 2 and quarters 1 and 2 were strong (.60<r<.80). The correlation for quarter 3 was moderate (r=.54). All correlations were significant at an alpha level of .01 (2-tailed).
|
|
|
|
| Discussion |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionAppendix.References |
|---|
The factor analysis of the professional behavior items suggested that each subdivision gives us some unique information and that items within a subdivision appear to be more highly correlated to each other than to items in other sections. These findings supported our use of a criterion-based scoring method for professional behaviors rather then adding the scores. In our experience, using the professional behavior section as a criterion-based evaluation makes our expectation for these criteria very clear to CIs as well as to students. We believe that all of the criteria should be met at all times regardless of the level of the clinical education experience.
The factor analysis of the patient management items clearly showed there was one distinct component, indicating that it was acceptable to combine all of the item scores into a single composite score. Every item in the patient management section loaded substantially on that component (all factor loadings for all items across all time periods were .74), indicating all of the items represent patient management skills. In examining the correlation matrix, all of the items were highly correlated with every other item, indicating that this is a homogeneous set of items. Based on coefficient alpha for the patient management scale, it is evident that all items on the scale consistently measure patient management.
External validity was demonstrated by the high correlation between the patient management scores and the global ratings of the students’ clinical competence. Clearly, the CIET is measuring a student's clinical competence. Higher patient management scores were associated with higher global ratings of clinical competence. In addition, the results of the repeated-measures ANOVA demonstrated that the CIET was measuring changes in the patient management scores as the students progressed in the clinical education sequence. In examining these data, it also was clear when students on the year-long affiliation progressed beyond "at that level," which we defined as basic competence to graduate. The CIET allowed the CIs to rate the students at a higher level.
We also demonstrated that the CIET was practical and easy to use based on the survey given to the CIs. Further anecdotal information and focus group interviews with our CIs since this survey indicate that they are able to complete this tool in a timely fashion. Clinical instructors reported that it takes between 30 and 60 minutes to complete the CIET compared with 2 to 3 hours for other clinical evaluation tools they have used. Most of our CIs also take students from other programs who use a different instrument, and they have reported to us that they prefer the CIET.
Limitations
Although in past evaluations of clinical tools, interrater reliability was determined, we did not feel that it was practical or appropriate in this case.13 Two CIs simultaneously examining a student does not match the real-life environment of the clinic where you see one CI working with one or more students. In addition, because student behaviors and performance are the culmination of observing multiple clinical encounters over an extended period of time, providing a situation in which 2 evaluators can observe student performance is impractical. Although we did not perform a test-retest reliability study, we believe that the responsiveness of the CIET is an indication of its reliability. The tool is able to differentiate students at various points in the curriculum. If test-retest reliability were poor, we would not expect the CIET to be able to differentiate among different points in the clinical education experience.
Use of the CIET in other academic settings may be limited, as we collected evidence for validating the tool only with students graduating from our program. We do feel that use of the CIET is generalizable to similar programs such as ours that exist in a large and diverse medical system. More than 100 CIs in a variety of settings used the tool to evaluate our students.
Future Plans
Based on the results of this study, individual items that were unclear on the CIET were reviewed and revised, and we plan to collect further data to provide validity evidence for this newer version. Other future plans include obtaining external validity evidence to determine whether the students’ scores on the CIET can predict performance on the National Physical Therapy Examination or clinical performance after graduation. In the future, we would like to have other academic programs use the CIET to demonstrate its generalizability to a variety of physical therapy clinical settings.
| Conclusion |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionAppendix.References |
|---|
| Appendix. |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionAppendix.References |
|---|
|
| Footnotes |
|---|
This work was presented as an education platform at the Combined Sections Meeting of the American Physical Therapy Association; February 23–27, 2005; New Orleans, La.
| References |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionAppendix.References |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
