Accreditation and Assessment

Measure 1. Completer Effectiveness (R4.1) 

COMPLETER IMPACT IN CONTRIBUTING TO P-12 STUDENT-LEARNING GROWTH:  As reported in AY20-21, school divisions across the Commonwealth closed schools in March 2020 to prioritize the health and safety of Virginia children and their teachers, and to lessen the spread of COVID. The majority of Virginia publics schools went fully remote for the Academic Year 2020-2021 as guidelines were being continuously revised based on new information and questions that arose.

In March 2021, the Virginia Department of Education released revised Guidelines for Uniform Performance Standards and Evaluation Criteria for Teachers. The Guidelines noted that the teacher performance standards cannot be modified; however, school divisions may modify the performance indicators and performance rubrics, and the approach and incorporation of observation, documentation, and other data sources. The measure of student progress (i.e., completer effectiveness) remains required to be a significant component of the evaluation; yet “how student academic progress is met in the evaluation is the responsibility of the local school division.”  Essentially all Virginia teachers are required to be evaluated against the standards, but how they are evaluated, and whether this evaluation is collected in a systematic, automated process is decided at the local level.

In preparation for AY2021-22, school divisions began to re-evaluate the COVID climate, trying to determine what best suits the safety and educational needs of their local community. Approaches ranged from fully remote, to blended, to remaining virtual.  As one employer commented in the VEAC Employer survey (more details below), “[xxx] is developing.  Most of her experience has been in atypical school years due to COVID.  I feel as we gain normalcy, she will continue to grow and develop.” There continued to be substantial COVID-related challenges as protocols for handling positive COVID cases changed throughout the school year.

During this same period, the Virginia Department of Education discontinued providing each EPP with a list of the completers and their place of employment (see below). This added another barrier for each EPP knowing which school district employed their completers.

As a stakeholder in Virginia Education Assessment Collaborative (VEAC), the EPP plans to collaborate with other EPPs to determine the most beneficial approach to collecting evidence of completers contribution to P-12 student-learning growth.

In spring 2023, the EPP reached out to multiple local school divisions to explore the possibility of collecting Teacher Evaluation data. This was modeled after what other VEAC EPPs have done/are currently doing. Not having done this prior to 2023, the EPP recognized the need to establish new relationships with the appropriate school division contacts and to establish relationships that are mutually beneficial to the EPP and the districts. The districts expressed that they were not able to provide the EPP with this information. The EPP has continued to seek out stronger partnerships with the districts in order to identify measures that can be used post-program completion to evaluate educator effectiveness. The long term goal is that once the EPP establishes a reporting system with this local school division, the EPP can collaborate with other EPP stakeholders in VEAC to collect Teacher Evaluation on all Virginia EPP completers in this school division and to share this information so that all Virginia EPP have equivalent, benchmarkable data as evidence for contributing to P-12 student-learning growth.

COMPLETER EFFECTIVENESS IN APPLYING PROFESSIONAL KNOWLEDGE, SKILLS, AND DISPOSITIONS: In the meantime, other VEAC EPPs have been working on piloting a standardized focus group protocol. The EPP plans to pilot the protocol during the Fall 2024 semester. The focus groups will utilize the VEAC focus group protocol and will provide program completers, school partners, and community members with the opportunity to discuss the EPPs program and its effectiveness.

As noted below, the EPP participates in the VEAC Completer Surveys. The EPP provides an in-house generated list of EPP completer emails to VEAC’s survey manager (University of Virginia) to send out and collect responses as evidence for CAEP R4.2, R4.3, RA4.1 and RA4.2.

During the AY22-23, the EPP worked to increase its rate of response by 1) providing more education and background information on the importance of the survey to EPP students who are about to be program completers, 2) having better non-EPP email address for this selected group collected through multiple measures including Program Completer Exit Surveys completed at licensure check-out meetings, and 3) having program faculty email this selected group in advance of the survey as a “heads up” notice (using a VEAC template email, of course!).

Within this Program Completer Exit Survey, program faculty will be asking for volunteers to participate in focus groups. VEAC has created focus group questions and protocol that will be initially piloted by James Madison University, and then by Mason. With the increase in zooming, remote focus groups also can be considered without the need for travel and additional time considerations. Planning is in place for implementation in mid fall 2024.

INITIAL LICENSURE (R4.2) 

VEAC Employer Satisfaction Data

As noted in the Initial Licensure Employer Survey VEAC Report, “for our 2022-2023 cycle, VEAC fielded the Employer Survey to employers of completers from 31 EPP Initial Standard 4 partners” (up from 29 EPPs the previous year). Upon closing the survey in April 2023, VEAC collected 1317 employer complete and partial responses (average response rate of 35%) and 1325 completer complete and partial responses (22% response rate). For the University of Mary Washington, the EPP had a 38% (n = 58) employer response rate which was down from last year’s 39% response rate. The EPP had a 25% (n = 38) completer response rate, which was up from the prior year’s 22% response rate. Response rates were based on the total number of contacts submitted to VEAC minus the number of failed/bounced emails. While this rate is higher than many other VEAC EPPs and the completer response rate has increased due to increased efforts to secure better contact information for completers, UMW plans to continue to use focused approaches to educating program participants on the survey and to collect accurate participant contact information.

The VEAC Employer Survey asked numerous questions about skills and overall satisfaction. Overall satisfaction was rated on a 5-point Likert scale with 1 = Not Ready and 5 = Fully Ready. Employers rated UMW program completers Fully Ready (44.44%), Mostly Ready (37.04%), Moderately Ready (16.67%), and Minimally Ready (1.85%). Across the larger VEACH survey, employers rated completers Fully Ready (54.47%), Mostly Ready (30.71%), Moderately Ready (10.18%), Minimally Ready (2.73%) and Not Ready (1.90%). Employers of UMW program completers were overall satisfied with their preparation (EPP mean = 4.24).

UMW program completers rated their overall satisfaction on their preparation as 4.39 on a 5-point Likert scale with rankings of “extremely dissatisfied (1), somewhat dissatisfied, neither satisfied nor dissatisfied, somewhat satisfied, or extremely satisfied (5)” (overall VEAC mean for completer self-rating satisfaction = 4.43).

Employers rated EPP program completers on 12 different questions related to things such as understanding of curriculum/content, planning, engagement, data informed decision making, resources and routines, ethics, monitoring student academic progress, use of technology, commitment to equity in practices, collaboration, assessment, and reflection. The questions were rated on a 4-point Likert Scale with “unacceptable (1), developing/needs improvement (2), proficient (3), and exemplary (4).” The average rating of all indices by employers was 3.0 (Proficient). The mean across all VEAC program completers was 3.26.

Program completers rated themselves on the same 12 questions related to things such as understanding of curriculum/content, planning, engagement, data informed decision making, resources and routines, ethics, monitoring student academic progress, use of technology, commitment to equity in practices, collaboration, assessment, and reflection. The questions were rated on a 4-point Likert Scale with “unacceptable (1), developing/needs improvement (2), proficient (3), and exemplary (4).” The average rating of all indices by completers was 3.08 (Proficient). The mean across all VEAC program completers was 3.28.

Program completers identified item “ID: Systematically gathering, analyzing, and using all relevant data to measure student academic progress, guide instructional content and delivery methods, and provide timely feedback to students, parents, caregivers, and other educators” as the area they felt the least prepared in (EPP mean 2.82, VEAC mean 3.06). This item was also the lowest scoring item across all program completers surveyed by VEAC.

Program completers identified item “IF: Maintaining a commitment to professional ethics, collaborating and communicating effectively, and taking responsibility for and participating in professional growth that results in enhanced student learning” as the are they felt the most prepared in (EPP mean 3.29, VEAC mean 3.49). This item was also the highest scoring item across all program completers surveyed by VEAC.

These reports were widely shared with the faculty to review and inform their program goals and improvement; however, a deep dive will be done to determine why certain UMW results are overall lower than the Virginia average. These VEAC surveys are critical in informing our licensure programs about how the preparation of our programs is perceived.

Stakeholder Involvement (R5.3): The EPP has a recently revised Quality Assurance System which provides programs with the opportunity to look at multiple measures to inform their program goals on an annual basis. In addition to the VEAC Completer and Employer Satisfaction Survey data, the EPP utilizes data gathered from feedback from an Advisory Board. The Advisory Board meets once per semester and includes representatives from local school district leadership. Additional stakeholders, like UMW students, UMW faculty, district mentor teachers, and community members are also invited to attend the Advisory Board meetings. Since they are made up of a variety of stakeholders and representatives, they bring multiple voices and perspectives to the table when discussing the licensure programs. The Advisory Board actionable feedback – formal and informal- informs all aspects of each licensure program and serves as a measure for our QAS, and as evidence of stakeholder involvement.

In addition to our EPP Advisory Board, as an active member in VEAC, the EPP is also collecting additional stakeholder feedback through the VEAC Surveys. The quantitative results, as well as the qualitative responses, of our completers and employers serves as some of the multiple measures for the EPP’s DARE, as evidence for informing the EPP’s mastery of Standard 4, and as evidence of stakeholder participation.

The EPP had a total of 101 candidates who successfully completed all of the state licensure requirements and were recommended for licensure.

One indicator of the candidate’s competency at completion of the initial licensure program is the EPP’s internal assessment tool, the Internship Evaluation Rubric.  The tool is based on the 10 InTASC standards, VDOE standards for teachers, and CAEP requirements for EPPs.  Each category has a rating from 1 to 4, with the target being a 3 in each indicator.  For the 2022-2023 academic year the UMW College of Education faculty and staff dove into the Internship Evaluation Rubric data using the recently adopted Data Dialog protocol. Items are scored on a scale of “not observed (0), developing (1), competent (2), and accomplished (3)” with the goal by the end of the internship for the candidate to score at least a 2/competent on each item. When looking at the data across all candidates, no items had an average score under 2. The faculty/staff identified the strengths of the candidates as being:

• Performance Indicator 3.1 The candidate collaborates with others to build a positive learning climate marked by respect, rigor, and responsibility. (InTASC 3; VDOE 6)
• Performance Indicator 4.2 The candidate creates learning experiences that make content accessible and meaningful for learners to assure mastery of the content. (InTASC 4, VDOE 1)
• Performance Indicator 9.3 The candidate practices the profession in an ethical manner. (InTASC 9, VDOE 7)

The faculty/staff identified the area with the most opportunity for growth to be:

• Performance Indicator 6.2 The candidate uses assessment to engage learners in their own growth. (InTASC 6, VDOE 4, 8)

This data is consistent with the data evaluated from the VEAC program completer and employer satisfaction surveys.

The EPP had a total of 379 candidates admitted and/or enrolled in the initial teacher certification or licensure.

VDOE’s annual IPAL report of EPP completer employment showed that 77.41% of EPP’s spring 2022 completers were employed in Virginia school districts for the 2022-2023 school year. Students who were not employed in a Virginia school district may have been employed in a Virginia private school or have accepted employment out of state. The VDOE’s IPAL report only reports data on in-state public school employment.

EPP internal tracking via the Program Completer Exit Survey of 2022-2023 program completers, identified 64.38% of the program completers had received Letters of Intent from public school districts, 5.48% had indicated they were licensing out of state, 1.37% had a signed letter of intent with a private school, and 28.77% had not yet signed a letter of intent anywhere as of the end of the semester of program completion, but prior to degree conferment.

The ongoing shortage of educator professionals in Virginia leads to many opportunities for employment of educators in Virginia (http://www.doe.virginia.gov/teaching/workforce_data/ ). It is important also to note that although the EPP is provided with limited employment data from VDOE, we make every effort to gather completer information.