STANDARD 2: ASSESSMENT SYSTEM AND UNIT EVALUATION
The unit has an assessment system that collects and analyzes data on the applicant qualifications, the candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.
2.1 Assessment System and Unit Evaluation
How does the unit use its assessment system to improve candidate performance, program quality and unit operations?
2a. Assessment System
The Teacher Education Unit (TEU) assessment system was developed to collect, analyze, and disseminate data based upon the unit's conceptual framework, university requirements, state standards, and program requirements of specialized professional associations (SPA). Data are collected at three transition points in the initial programs and at four transition points for the advanced program. The conceptual framework, Teacher as Decision Maker, is aligned with the 2011 InTASC standards and current Oklahoma State Standards (adopted June 2014) and supports the mission of Langston University (LU). It reflects the knowledge, competencies, and professional dispositions candidates demonstrate in portfolios. The 2014-2015 PASS-PORT Undergraduate Artifacts Master List identifies required signature artifacts in designated courses, aligned to InTASC standards. Currently PASS-PORT serves as the online portfolio management system for the collection, analysis, and dissemination of TEU data. Assessments submitted to SPAs as outlined in program reviews, are aligned to the individual content areas, and are anticipated to be included in PASS-PORT by fall 2016. Candidates are evaluated by internal and external assessments at each transition point. Candidates collect specific artifacts in professional education courses, upload them to PASS-PORT, and submit them to their course instructor, unless noted otherwise on the Master List. For example, to advance from Transition Point 1, or Portal 1,
to Portal 2, candidates complete the course ED2212 Historical and Philosophical Foundations of American Education and the three signature artifacts embedded in the course: Philosophy of Education, Disposition (self assessment), and Disposition (evaluated by the instructor). The philosophy paper is one element included in the portfolio evaluation and is assessed using a three-level rubric. Candidates receive directions, guidelines, and training on each signature artifact through course syllabi, course instructors, and assessment-specific directions and rubrics.
At the initial and advanced levels, multiple measures are used to assess candidates' content knowledge, pedagogical content knowledge, professional and pedagogical knowledge and skills, student learning, and professional dispositions. The assessment system is monitored and assessments are added, deleted, or revised based upon results of data analysis and decision made by the Teacher Education Council (TEC). TEC membership, including faculty from the School of Arts and Sciences and the School of Agriculture and Applied Sciences, is representative of the LU professional community. The TEC is the major curriculum review body for the TEU. It provides a mechanism for and facilitates collaboration between TEU faculty and other faculty at LU involved in the preparation of professional educators. The TEC serves in a decision-making capacity on all matters pertaining to teacher education within the TEU with the Dean having the final authority on all matters. The TEU adheres to the rules and regulations set forth for teacher certification and licensure by the Office of Educational Quality and Accountability (OEQA). For programs outside of the School of Education and Behavioral Sciences (SoEBS), Early Childhood Education and Secondary, the TEC serves as the coordinating body with the Director of Teacher Education, Certification, and Field Experiences responsible for the dissemination of information to the individual program chairs or department heads. At the advanced level, under the direction of the Dean, all matters pertaining to program and degree options are administered by the Director of the Graduate Program, who brings all such requests and concerns before the Graduate Program Commission and ensures adherence to the OEQA rules and regulations for Other School Personnel.
The TEU monitors faculty performance as part of the assessment system utilizing the Faculty Professional Performance Plan (F3P) Evaluation of Teaching, Scholarship, and Service and LU Student Course Evaluations. The F3P outlines a systematic and comprehensive process of faculty evaluation designed to assess the professional performance of faculty in the areas of teaching effectiveness, scholarship, and service. The process allows for self-evaluation and input from candidates, colleagues, and administrators. This system provides information for decision-making concerning employment, promotion, and tenure. Faculty submit artifacts and work samples as evidence of individual performance related to the contents of the F3P.
2b. Data Collection, Analysis, and Evaluation
Between 2011-2013, the SoEBS underwent substantial personnel changes with the Dean; Director of Teacher Education, Certification, & Field Experiences; Coordinator of Assessment & Student Advisement; and several senior faculty retiring or leaving the unit. The TEU was without an assessment coordinator until a Unit Assessment Coordinator was appointed in fall 2013 and renamed the Director of Unit Assessment and Accreditation in fall 2014. The Director oversaw the initial adoption (fall 2013) of the electronic portfolio system, PASS-PORT; assumed responsibilities for combining the existing paperbased assessment system with the newly adopted system; led full implementation of PASS-PORT (fall 2014) for all professional education courses; collected, disseminated, and reported unit data; and provided faculty and candidate training and support.
PASS-PORT allows candidates to collect and submit artifacts, or work products, and for faculty to monitor and assess candidate performance at each transition point, or portal. PASS-PORT facilitates the TEU's capacity to collect, analyze, and disseminate data based on the evaluation of outcomes and signature artifacts through the use of standards-based rubrics. Candidates have access to the instructor's evaluation on set criteria outlined in the rubrics and are required to attain a passing score on each artifact within a portal prior to moving to the next portal. PASS-PORT is designed to serve as a feedback loop that provides data to support candidate growth and program/unit decision-making.
To ensure implementation, consistency, and follow-through in utilizing PASS-PORT, in-person trainings have been provided on all Langston campuses, "How to" videos for faculty and candidates have been developed and made available, and assessment system updates for faculty are provided routinely at TEC meetings. Signature artifacts are evaluated using uniform rubrics to support the collection of reliable data across programs. The Disposition Evaluation serves as an example of a uniform instrument that was developed to assess candidate dispositions at multiple transition points. The Disposition Evaluation was approved in fall 2014 and will provide data for one application across program levels in spring 2015.
Due process procedures are listed in the Student Handbook; records of resolutions and documentation of complaints will be available to team members during the onsite visit.
2c. Use of Data for Program Improvement
Prior to fall 2014, data were not consistently collected and applied electronically; thus, communication of this data to existing and new faculty and candidates could be described as unsystematic. Pockets of artifacts, portfolios, and data existed in paper form but the assessment system as an organized and disciplined process for collecting, reporting, and analyzing unit-wide data was ineffective. Established procedures and timelines used to ensure faculty and candidate participation were unsuccessful in gathering and maintaining data appropriate for decision-making and program improvement at the unit level. In revitalizing the assessment system, PASS-PORT serves as the beginning steps of establishing a systematic and effective assessment system that provides the TEU, faculty, candidates, and stakeholders access to data that can be routinely disseminated and used for evaluating the effectiveness of courses, programs, and clinical experiences.
Programs provided examples of collection and use of data to improve candidate performance and the overall program in Section V of the program reports submitted to their respective SPAs. For example, Early Childhood Education identified (1) developing stronger constructed response opportunities in content coursework in light of candidate performance below desired levels on the state content exam, (2) increasing candidate opportunities for critical thinking activities, and (3) continuing to provide opportunities to develop strong advocacy skills as program objectives based upon data. The Special Education faculty also identified improving candidate performance on the constructed response sections of the state exams as an area needing attention. Of the seven programs submitted for national review, Early Childhood Education, Elementary Education, Special Education, and Science Education (Biology/Chemistry) were fully recognized; Health, Physical Education, and Recreation (HPER) along with Mathematics Education were recognized with conditions; and English/Language Arts (ELA) Education was recognized with probation. HPER & Math Education will resubmit in response to conditions & ELA will submit a new program review in spring 2015. The Instrumental & Vocal Music programs, without a national SPA, will submit program reviews in response to conditions at the state level in spring 2015.
2.2 Moving Toward Target or Continuous Improvement
Please respond to 2.2.a if this is the standard on which the unit is moving to the target level. If it is not the standard on which you are moving to the target level, respond to 2.2.b.
2.2.a Standard on which the unit is moving to the target level
Describe areas of the standard at which the unit is currently performing at the target level for each element of the standard. Summarize activities and their impact on candidate performance and program quality that have led to target level performance. Discuss plans and timelines for attaining and/or sustaining target level performance as articulated in this standa
n/a
2.2.b Continuous Improvement
Summarize activities and changes based on data that have led to continuous improvement of candidate performance and program quality. Discuss plans for sustaining and enhancing performance through continuous improvement as articulated in this standard.
Plans for sustaining and enhancing performance through continuous improvement include all of the information contained in 2.3 Areas for Improvement and the following:
The TEU is reviewing and revising unit processes and procedures specific to data collection and dissemination. PASS-PORT, an electronic online data management system, was selected in spring 2014, piloted in fall 2014, and fully implemented in spring 2015. The unit is transitioning from a paper-based assessment system to PASS-PORT in order to streamline the data collection process and dissemination of data to inform program/unit decision-making. PASS-PORT is being used at the unit level with plans to use PASS-PORT at the program level by fall 2016. PASS-PORT has the capacity to provide the TEU regular and comprehensive information on applicant qualifications, candidate proficiencies, competence of graduates, unit operations, and program quality.
The TEU is in the process of transitioning from a paper-based assessment system to PASS-PORT and to ensure transparency of unit assessment. Assessment processes and procedures along with assessment instruments and handbooks (e.g. PASS-PORT Handbook and Navigating Teacher Education) are being reviewed, and as necessary updated/revised, to ensure alignment of the conceptual framework (revised and approved spring 2014) and professional dispositions delineated in institution, state, and national standards, specifically InTASC. The goal of reviewing and refining existing assessment instruments is to improve fairness, accuracy, and consistency of the unit's multiple assessment instruments, assessment procedures, and unit operations to allow for the regular and systematic collection compilation, aggregation, summarization, and analyses to improve candidate performance, program quality, and unit operations. Admission interviews, field experiences, and clinical teaching instruments are priorities in the reviews to ensure consistency of scales, the alignment of criteria with standards, use of verbiage that are observable and measureable, and ensure consistent formatting. These instruments will be complete and ready for implementation beginning fall 2015. On-going review of the instruments will be part of the Curriculum Committee and Advisory Committee at the close of each semester.
Timelines for data collection and analysis related to candidates and unit operations are being refined to ensure PASS-PORT provides for the systematic use of data to evaluate the efficacy of courses, programs, and clinical experiences. PASS-PORT will allow candidate assessment data to be regularly shared with faculty and used to evaluate and make improvements in the unit, and its programs, courses, teaching, and field and clinical experiences. Additionally, opportunities for review of bias and fairness are made available through PASS-PORT as it allows for the analyses of results of internal and external assessments based upon similar competencies. Elimination of bias will also be accomplished, in part, by utilizing PASS-PORT to allow multiple assessors to assess the same artifact, providing for the examination of inter-rater reliability.
PASS-PORT rubrics are designed to assess the required signature artifacts (aligned to InTASC 2011) and the Disposition Evaluation (adopted spring 2014) are aligned to the revised conceptual framework (approved spring 2014), which is aligned to state and national standards. These rubrics are available for download through Desire2Learn (D2L) and PASS-PORT. The rubrics embedded in PASS-PORT and associated with signature artifacts allow for the immediate collection of data upon completion of the rubrics.
Beyond PASS-PORT, TEU faculty and program coordinators are in the process of aligning course objectives, assessments, rubrics, and syllabi with revised Specialized Professional Association (SPA) standards. Faculty are also working to ensure consistency across campuses regarding course content, textbook, and syllabi for courses taught on more than one campus.
The TEU has adopted a continuous improvement process to allow for the on-going review and, as necessary, revision of unit processes and procedures specific to data collection and dissemination. An annual Program Quality Internal Review (PQIR) process, implemented in spring 2015, is used to ensure quality evidence is collected and analyzed in a systematic manner and used to inform and improve unit and program operations. The PQIR utilizes data from PASS-PORT, surveys, course evaluations, and faculty evaluations to assist programs in identifying program strengths and weaknesses, identifying areas for concern, creating a timeline for addressing areas of concern, and outlining the progress toward meeting the previous year's PQIR areas of concern. The PQIR, a living document of continuous improvement, focuses on identifying candidate learner outcomes, collecting data related to the learning outcomes, and analyzing the results in order to recommend changes to academic programs and the unit.
All activities listed above demonstrate the unit's commitment to continuous improvement in the area of assessment. The governance now in place is capable of sustaining the continuous improvement process using PASS-PORT, faculty involvement in the development and evaluation of assessment tools, increased documentation of assessment processes and procedures, and the availability of data. The increased focus on assessment within the unit and the planned expansion of program-specific use of PASS-PORT assists in the faculty's internalization of the necessity and method to collect reliable and bias-free evidence to use in making informed, data-driven decisions, and in moving the TEU toward a culture of assessment.
2.3 Areas for Improvement Cited in the Action Report from the Previous Accreditation Review
Summarize activities, processes, and outcomes in addressing each of the AFIs cited for the initial and/or advanced program levels under this standard.
Standard 2 had two AFIs cited in the previous accreditation review related to initial and advanced levels: • The unit's assessment system was developed with limited involvement of the professional community. • The assessment system does not provide for the regular analysis of data to improve unit operations.
AFI Addressed: The unit's assessment system was developed with limited involvement of the professional community.
Currently the TEC is in the process of developing a constitution and bylaws. At present, formal policies and procedures do not exist to govern its operation. Following, the completion and approval of the constitution and bylaws, faculty from outside the unit, alumni, and candidates will be invited and encouraged to serve as part of the TEC.
An Advisory Council was established in spring 2015 to serve as an advisory and advocacy network for the TEU through a regular exchange of information and ideas. The Advisory Council is led by the Dean and meets once each semester to review program and unit data as well as to provide feedback that may lead to program and/or unit improvement. This Council facilitates communication between the TEU, local leaders, and public school educators to review data, policies, and procedures to ensure teacher education graduates develop the skills, knowledge, and personal dispositions required for professional success.
AFI Addressed: The assessment system does not provide for the regular analysis of data to improve unit operations.
The unit assessment system is undergoing significant changes regarding the collection, review, and distribution of all assessment data. Following the last accreditation review, the SoEBS underwent substantial faculty changes. The turnover is especially noteworthy regarding the position of Dean; filled by four individuals, the longest term was held by an Interim Dean who served three years prior to the appointment of Dr. Ruth Jackson in January 2014. As a result of these changes, specifically the changes in leadership, many standard assessment processes, requirements, and protocols were misinterpreted or minimally followed in maintaining the day-to-day operations of the unit.
The Coordinator of Assessment and Student Advisement position was created in 2007 to develop and maintain a centralized data collection system for the unit compatible with the university's data system. The Coordinator was responsible for implementing the assessment system. Coordinator tasks included gathering, organizing, interpreting, analyzing, summarizing, and presenting unit data. The Coordinator was charged with making the data readily available for faculty use in decision-making and program improvement. The objectives of the assessment system were to provide the unit with a cohesive way to gauge the relationship of unit assessment activities in relation to the vision, mission, core values, and expected learning outcomes of the university while adhering to state and national accreditation data collection requirements. However, between 2007 and 2013 a dual system was in place that allowed candidates to create an online portfolio utilizing the web-based software application LiveText or to create a hard-copy, three-ring binder portfolio. There were inconsistencies in expectations and requirements related to artifact collection and storage as some candidates purchased LiveText subscriptions and some created hard-copy portfolios. The Coordinator, maintaining a centralized data collection system, was the only faculty able to access LiveText for evaluation purposes. Following his resignation in spring 2012, faculty were unable to support the use of LiveText as they did not possess the training or capability to assess items submitted to LiveText. In fall 2013, a Coordinator of Unit Assessment was appointed to fulfill the goals of the position established in 2007. The following fall, the Coordinator position was replaced with the position of Director of Unit Assessment and Accreditation, at which time the unit began transitioning from a paper-based assessment system with binder portfolios to an electronic assessment system. PASS-PORT was selected in fall 2013 as the online portfolio management system that would be used across the TEU. Signature artifacts were modified from existing portfolio assignments or newly developed in alignment with the 2011 InTASC standards. Between fall 2013 and spring 2015, candidates' completed artifacts and/or rubrics were maintained in files in the office of the Director of Teacher Education, Certification, and Field Experiences. The scores assigned to the artifacts were entered into Excel spreadsheets by the Director of Unit Assessment and Accreditation to allow for the aggregation and disaggregation of data. Beginning spring 2015, PASS-PORT will serve as the primary storage for artifacts and assessments and provide the opportunity to generate reports without the need to maintain hard copies or enter data in Excel spreadsheets. It is the goal of the TEU to use PASS-PORT to collect program data, as well as unit data, beginning Fall 2016. Prior to implementation, signature artifacts and rubrics need to be reviewed and assessed for validity, reliability,and bias; alignment between artifacts and current standards needs to be ensured; and platforms within PASS-PORT need to be created. Until the system is fully electronic on a program and unit level, hard copies for current artifacts will continue to be collected, the Director of Unit Assessment and Accreditation will continue to collect the data from the artifacts at the end of each semester, and utilize Excel to maintain electronic records that lend themselves to dissemination and review.
2.4 Exhibits
2.4.a Description of the unit's assessment system including the requirements and key assessments used at transition points
2.4.a Advanced Assessment Table
2.4.a SoEBS Compiled Outcomes Asst 10.31.14
2.4.a Advanced Level Program Outcomes Asst 2013-2014
2.4.a Educational Leadership Portfolio Handbook
2.4.a Advanced Transition Levels
2.4.a Clinical Teaching Handbook
2.4.b Admission criteria and data from key assessments used for entry to programs
2.4.b Graduate Student Handbook
2.4.b Unit Data: Admissions Interviews
2.4.b Unit Data: Field Experience Evaluations by University Supervisors
2.4.c Policies, procedures, and practices for ensuring that key assessments of candidate performance and evaluations of program quality and unit operations are fair, accurate, consistent, and free of bias
2.4.d Policies, procedures, and practices for ensuring that data are regularly collected, compiled, aggregated, summarized, analyzed, and used for continuous improvement
2.4.e Policies, procedures and practices for managing candidate complaints
2.4.f File of candidate complaints and the unit's responses and resolutions (This information should be available during the onsite visit)
2.4.g Examples of significant changes made to courses, programs, and the unit in response to data gathered from the assessment system
2.4.g Plans of Study (SPED, ELE, HPER - others available in Standard 1)