Model for development

This article describes a model for survey development used by the Department of Physical Therapy at Saint Louis University to assess the extent to which the program prepares students for a career as physical therapists. From a review of departmental documents including mission and philosophy statements and curriculum goals, eight performance outcomes (e.g., competence as a general practitioner, leadership, ethics, attitude for service, etc.) and two program characteristics (curricuum and program atmosphere) were determined. External documents including accreditation criteria for graduate performance were also reviewed and found to he consistent with the identified characteristics. Based on these findings, items for three surveys were developed and assessed. A 1-year alumni survey solicited feedback on the graduates' perceptions of their educational preparedness for clinical practice. The graduates' employers then were surveyed for an assessment of the graduates' job performance. A 3-year alumni survey followed up to reassess the graduates' perceptions of their educational preparedness and to track the alumni's activities since graduation. Other programs in physical therapy and other health care disciplines can use this model to develop valid survey instruments to assess their program's effectiveness. J Allied Health. 2003; 32:179-184.

PRESSURE ON EDUCATIONAL institutions to show the effectiveness of their programs has increased since the 1990s. State legislators, parents, and students want assurances that students will obtain the education promised.1 Accreditation agencies also demand that institutions and programs implement outcome assessments to ensure that students have opportunities to achieve the educational goals and become competent in their chosen field.2 The evaluative criteria that must be met for accreditation by the Commission on Accreditation of Programs for Physical Therapy Education include program assessment, which encompasses assessment of student learning outcomes

Methods for conducting outcome assessments range from the use of commercially available achievement and personality tests to instruments that are institutional specific.4 Much of the literature on student outcome assessment focuses on the development of a university-wide program of assessment4-6 When alumni are surveyed, the outcomes of interest frequently are related to overall satisfaction with the college experience and successful job placement.4,6 When Northeast Missouri State University changed its mission from one of primarily preparing teachers to being a small multipurpose university, they wanted to determine how successful they were in graduating individuals who were nationally competitive in many diverse fields. Follow-up graduate surveys were used to determine the type of job placements the graduates obtained.5 When information at the individual program level is desired, one suggestion is to put short inserts into the university or college survey.6 We believe that student outcome assessment at the program level requires the same thought and effort that is recommended for university-wide or college-wide assessment programs.

Student outcomes assessment should include several components, occur at multiple points, and include an appraisal of the educational program from different points of view.7 A review of the curriculum can include regular assessment of curriculum prerequisites, the content and sequence of courses, and monitoring of student learning opportunities outside the program. Assessment of student academic progress and professional development should occur at designated times in the program. For educational programs preparing students for health professions, assessments of practicum or clinical experiences provide opportunities to determine the student's readiness for work in the profession. Students and clinical instructors have different points of view, and an analysis of student clinical performance ratings and clinical instructor perceptions of the program based on student performance during clinical rotations can provide useful information. Finally, analysis of alumni perceptions of the program and their reports of professional activities since graduation and analysis of employer perceptions of alumni performance provide views of the program after the alumni have been working in their fields. Although academic performance can be used to track student performance during the didactic portion of the curriculum, assessments of students' clinical performance and performance after graduation require other methods. Surveys are recommended as one way to gather these data.3 Without a clear plan for survey development, implementation, and use of results, however, the data that are gathered will not meet the needs of the program.6

The purpose of this article is to describe a model for developing survey instruments to assess the extent to which alumni of allied health programs show desired outcomes and to illustrate the use of survey information for curricular assessments. Although implementation of the model at Saint Louis University, Department of Physical Therapy, is used to illustrate the process; other allied health programs can use this model for developing and implementing student outcome assessments. Because the results of our surveys cannot be generalized to other programs, we have not included detailed results. We have included, however, some examples to show how we use the data.

Development of the Surveys

Outcome measurements are more meaningful and valid when program mission, philosophy, and goals are considered during development of the tools. Our first step was to review internal documents that described the characteristics we believed necessary for a physical therapist. The internal documents we reviewed included the department mission and philosophy statements and the department curriculum goals. From this review we identified eight performance outcomes expected of our graduates and two program characteristics related to the program. The performance outcomes are competence as general practitioner, leadership, ethics, attitude for service, critical thinking, effective communication, personal and professional growth, and management skills. The two program characteristics are curricular issues, such as sequencing of courses, and program atmosphere, such as providing a supportive learning environment.

Important external documents should be reviewed to improve further the validity and usefulness of the data. Because allied health programs undergo assessment by external accreditation bodies, the documents used to determine accreditation status should be reviewed. In addition, when they exist, discipline-specific documents used to assess student clinical performance should be reviewed. The external documents we reviewed included the Evaluative Criteria for Accreditation of Education Programs for the Preparation of Physical Therapists3 and the Physical Therapist Clinical Performance Instrument.8 Program curriculum goals should be compared with accreditation statements to determine the amount of congruency between the documents. When there is adequate agreement between the documents, the program can use the program curriculum goals to develop and review survey items. When we compared the Evaluative Criteria for Accreditation of Education Programs for the Preparation of Physical Therapists3 with our curriculum goals, the performance outcomes, and the program characteristics we found that they were consistent with one another. We did not compare directly the survey items with the Evaluative Criteria for Accreditation of Education Programs for the Preparation of Physical Therapist.3 Survey items were compared with the Physical Therapist Clinical Performance Instrument.8 Figure 1 illustrates the model we used to develop and assess our surveys.

To assess alumni performance comprehensively in relation to our curriculum and the characteristics we believed necessary for a physical therapist, we decided that three surveys were needed: a 1-year alumni survey, an employer survey, and a 3-year alumni survey. The purposes of the surveys were to determine the extent to which graduates are meeting the educational goals specific to our program, to identify improvements needed to meet program goals better and current requirements of practice, and to meet program and institutional standards for accreditation. When developing the surveys, we considered whether items were consistent with the purposes of a particular survey and whether an item was appropriate for the intended respondent (i.e., alumnus or employer). The university institutional review hoard approved all three surveys and the survey method.