Accreditation and Assessment

COMPLETER IMPACT IN CONTRIBUTING TO P-12 STUDENT-LEARNING GROWTH:  As reported in AY20-21, school divisions across the Commonwealth closed schools in March 2020 to prioritize the health and safety of Virginia children and their teachers, and to lessen the spread of COVID. The majority of Virginia publics schools went fully remote for the Academic Year 2020-2021 as guidelines were being continuously revised based on new information and questions that arose.  

In March 2021, the Virginia Department of Education released revised Guidelines for Uniform Performance Standards and Evaluation Criteria for Teachers. The Guidelines noted that the teacher performance standards cannot be modified; however, school divisions may modify the performance indicators and performance rubrics, and the approach and incorporation of observation, documentation, and other data sources. The measure of student progress (i.e., completer effectiveness) remains required to be a significant component of the evaluation; yet “how student academic progress is met in the evaluation is the responsibility of the local school division.”  Essentially all Virginia teachers are required to be evaluated against the standards, but how they are evaluated, and whether this evaluation is collected in a systematic, automated process is decided at the local level.  

In preparation for AY2021-22, school divisions began to re-evaluate the COVID climate, trying to determine what best suits the safety and educational needs of their local community. Approaches ranged from fully remote, to blended, to remaining virtual.  As one employer commented in the VEAC Employer survey (more details below), “[xxx] is developing.  Most of her experience has been in atypical school years due to COVID.  I feel as we gain normalcy, she will continue to grow and develop.” There continued to be substantial COVID-related challenges as protocols for handling positive COVID cases changed throughout the school year.  

During this same period, the Virginia Department of Education discontinued providing each EPP with a list of the completers and their place of employment (see below). This added another barrier for each EPP knowing which school district employed their completers.  

Based on the continued COVID challenges, and the lack of supporting documentation from the Virginia Department of Education, in AY2021-2022, the EPP did not contact local school divisions to collect anonymized and aggregated Teacher Performance Evaluation data on EPP completers that would serve as evidence that contributes to P-12 student learning growth. The EPP felt that the request would be too burdensome for the local school divisions, and that, due to the potential range in teacher evaluation implementation, the data may not be valid or reliable for ongoing program evaluation.  

As a stakeholder in Virginia Education Assessment Collaborative (VEAC), the EPP plans to collaborate with other EPPs to determine the most beneficial approach to collecting evidence of completers contribution to P-12 student-learning growth. 

In spring 2023, the EPP reached out to multiple local school divisions to explore the possibility of collecting Teacher Evaluation data. This was modeled after what other VEAC EPPs have done/are currently doing. Not having done this prior to 2023, the EPP recognized the need to establish new relationships with the appropriate school division contacts and to establish relationships that are mutually beneficial to the EPP and the districts  

The long term goal is that once the EPP establishes a reporting system with this local school division, the EPP can collaborate with other EPP stakeholders in VEAC to collect Teacher Evaluation on all Virginia EPP completers in this school division and to share this information so that all Virginia EPP have equivalent, benchmarkable data as evidence for contributing to P-12 student-learning growth. 

COMPLETER EFFECTIVENESS IN APPLYING PROFESSIONAL KNOWLEDGE, SKILLS, AND DISPOSITIONS: In the meantime, other VEAC EPPs have been working on piloting a standardized focus group protocol. The EPP plans to pilot the protocol as soon as it is available. The EPP plans to collect information for its licensure programs in groups, not all licensure programs at once, to make each collection more useful and targeted.  

As noted below, the EPP participates in the VEAC Completer Surveys. The EPP provides an in-house generated list of EPP completer emails to VEAC’s survey manager (University of Virginia) to send out and collect responses as evidence for CAEP R4.2, R4.3, RA4.1 and RA4.2.  

During the next AY22-23, the EPP plans to increase its rate of response by 1) providing more education and background information on the importance of the survey to EPP students who are about to be program completers, 2) having better non-EPP email address for this selected group collected through multiple measures including Program Completer Exit Surveys completed at licensure check-out meetings, and 3) having program faculty email this selected group in advance of the survey as a “heads up” notice (using a VEAC template email, of course!).   

Within this Program Completer Exit Survey, program faculty will be asking for volunteers to participate in focus groups. VEAC has created focus group questions and protocol that will be initially piloted by James Madison University, and then by Mason. With the increase in zooming, remote focus groups also can be considered without the need for travel and additional time considerations. Planning is in place for implementation in late spring 2023, and the EPP hopes to begin piloting using the protocol late fall 2023 or early spring 2024.  

As noted in previous Annual Reports, in the Commonwealth of Virginia, there is no clear mechanism for collecting and sharing data across the state education agency, EPPs, and P-12 school divisions. Collection of accurate emails and post-graduation contact information is a continuous challenge. Additionally, in summer 2021, the EPP also learned of the retirement of a critical, long-term Virginia Department of Education senior official, as well as a long serving, loved VDOE staff member who provided each Virginia EPP their completer list during summer months. Both of these changes greatly impacted the EPP’s ability to provide Virginia Education Assessment Collaborative (VEAC) (https://projectveac.org  ) with accurate completer lists and emails for completer satisfaction surveys.   

Despite these challenges, VEAC distributed and collected survey information for 29 Virginia EPPs for the third year in AY 2021-2022.   

Employer Satisfaction Data  

As noted in the Initial Licensure Employer Survey VEAC Report, “for our 2021-2022 cycle, VEAC fielded the Employer Survey to employers of completers from 29 EPP Initial Standard 4 partners” (up from 27 EPPs the previous year). Upon closing the survey in August 2022, VEAC collected 1169 employer complete and partial responses (29% response rate) and 874 completer complete and partial responses (29% response rate). For the University of Mary Washington, the EPP had a 39% (n = 65) employer response rate and a 22% (n = 41) completer response rate on the VEAC Surveys based on the total number of contacts submitted to VEAC minus the number of failed/bounced emails. While this rate is higher than many other VEAC EPPs, the UMW plans to use focused approaches to educating program participants on the survey and to collecting accurate participant contact information.  

The average rated readiness of the EPP completers by employers was 4.37 (on a 1-5 scale); this score was lower than the overall VEAC score of 4.43, and reflected the EPP’s overall results in the VEAC Employer survey. UMW completers scored higher than the overall VEAC group on 8 of the 14 items (items A, B, C, H, I, K, L, and M) and they scored equal to the overall VEAC group on 1 item (Item N). UMW completers scored lower than the overall VEAC group on 5 items (Items D, E, F, G, and J). These items related to use of data (overall VEAC = 3.17, EPP = 3.09), implementation of resources and routines (overall VEAC = 3.30, EPP = 3.28), maintaining professional ethics/communicating effectively/taking responsibility for professional growth (overall VEAC = 3.36, EPP = 3.34), work resulting in acceptable and measurable student progress (overall VEAC = 3.23, EPP = 3.22), and inclusion of multiple perspectives (family, community, etc.) into instruction (overall VEAC = 3.24, EPP = 3.23). The EPP plans to conduct an event where licensure programs can deeply review this data to understand how improvements can be made at the program level to preemptively support novice teachers and increase UMW program completer’s scores.  

Looking at the VEAC Employer Data in conjunction with the VEAC Completer data, for UMW, the average overall satisfaction of UMW completers scored was higher than the overall VEAC score (overall VEAC = 4.49, EPP = 4.51) but lower than the score last year (2020-2021 EPP = 4.86, 2021-2022 EPP = 4.51).  UMW completers scored measurably lower on Item H, “Selects technologies, informed by research, to promote learning for all students” (overall VEAC = 3.23, EPP = 3.0). This is a rating the program faculty need to review, and then reflect on how each program is preparing candidates for this measure.  

Three items were consistently scored lower for EPP completers across both the Completer Survey and the Employer Survey. These items need to be critically evaluated by each program. They are as follows: 

Item D: Systematically gathers, analyzes, and uses all relevant data to measure student academic progress, guide instructional content and delivery methods, and provide timely feedback to both students and parents throughout the school year. 

Completer EPP Mean: 3.02 vs VEAC Mean: 3.06 

Employer EPP Mean: 3.09 vs VEAC Mean: 3.17 

Item E: Uses resources, routines, and procedures to provide a respectful, positive, safe, student-centered environment that is conducive to learning.  

Completer EPP Mean: 3.34 vs VEAC Mean: 3.39  

Employer EPP Mean: 3.28 vs VEAC Mean: 3.30 

Item F: Maintains a commitment to professional ethics, communicates effectively, and takes responsibility for and participates in professional growth that results in enhanced student learning.  

Completer EPP Mean: 3.54 vs VEAC Mean: 3.56 

Employer EPP Mean: 3.34 vs VEAC Mean 3.36 

Many successes were identified through the open-ended responses of the Employer Satisfaction Survey. One teacher was identified as having been named her school’s New Teacher of the Year last year and was identified and recognized for her ability to provide “provide professional development to other educators” and because she “allowed other new teachers to come observe her class for their own learning.” Another teacher was identified as having been named their school’s Teacher of the Year this year for her “relationships with students and stakeholders and her continued dedication to meeting the needs of each and every child.”  Other teachers were identified for their reflective nature, ability to give quick feedback to students, and outright desire to collaborate with other colleagues. Areas of growth included general inexperience in handling certain situations with maturity, needed support with classroom management, and the need for support with “making the shift from teaching in a hybrid/virtual learning environment to full, in-person teaching.” 

These reports were widely shared with the faculty to review and inform their program goals and improvement; however, a deep dive will be done to determine why certain UMW results are overall lower than the Virginia average. These VEAC surveys are critical in informing our licensure programs about how the preparation of our programs is perceived.  

Stakeholder Involvement (R5.3): The EPP has a well-established Quality Assurance System which provides programs with the opportunity to look at multiple measures to inform their program goals on an annual basis. In addition to the VEAC Completer and Employer Satisfaction Survey data, the EPP utilizes data gathered from feedback from an Advisory Board. The Advisory Board meets once per semester and includes representatives from local school district leadership. Additional stakeholders, like UMW students, UMW faculty, district mentor teachers, and community members are also invited to attend the Advisory Board meetings. Since they are made up of a variety of stakeholders and representatives, they bring multiple voices and perspectives to the table when discussing the licensure programs. The Advisory Board actionable feedback – formal and informal- informs all aspects of each licensure program and serves as a measure for our QAS, and as evidence of stakeholder involvement.  

In addition to our EPP Advisory Board, as an active member in VEAC, the EPP is also collecting additional stakeholder feedback through the VEAC Surveys. The quantitative results, as well as the qualitative responses, of our completers and employers serves as some of the multiple measures for the EPP’s DARE, as evidence for informing the EPP’s mastery of Standard 4, and as evidence of stakeholder participation. 

One indicator of the candidate’s competency at completion of the initial licensure program is the EPP’s internal assessment tool, the Internship Evaluation Rubric.  The tool is based on the 10 InTASC standards, VDOE standards for teachers, and CAEP requirements for EPPs.  Each category has a rating from 1 to 4, with the target being a 3 in each indicator.  For the 2021-2022 academic year, 97% of candidates scored an average of 3 or higher across all indices. The average score was 3.24.  Particular areas of strength included: the candidate’s ability to understand the tools of inquiry and structures of the discipline (Avg score 3.85) and the candidate’s ability to organize and manage face-to-face and/or virtual environments that support individual and collaborative learning (Average score 3.43). 

The EPP had a total of 380 candidates admitted and/or enrolled in the initial teacher certification or licensure.  

EPP internal tracking via the Program Completer Exit Survey of 2020-2021 program completers, identified 68.3% of the program completers had received Letters of Intent from districts in the semester of program completion but prior to degree conferment.  

The ongoing shortage of educator professionals in Virginia leads to many opportunities for employment of educators in Virginia (http://www.doe.virginia.gov/teaching/workforce_data/ ). It is important also to note that although the EPP is provided with limited employment data from VDOE, we make every effort to gather completer information.