A data-driven program with a rich culture of assessment, the Physician Assistant program at NAU implemented ExamSoft to gain performance insights for multiple learning outcomes and educational objectives. With the help of ExamSoft’s reporting tools, the Physician Assistant program has been able to analyze programmatic performance by category, improve the quality of exam content, and identify valuable opportunities for remediation and retention.
Established in partnership with the University of Arizona, Northern Arizona University’s Phoenix Biomedical Campus educates students seeking degrees in Athletic Training (AT), Occupational Therapy (OT), Physical Therapy (PT), and Physician Assistant Studies (PA).
The PA program at Northern Arizona University (NAU) offers a Master of Physician Assistant Studies (MPAS) and is Arizona’s only PA program offered by a public university. The first year of the program is devoted to a didactic curriculum, while the second year consists of clinical rotations that provide hands-on experience in rural and community clinics throughout Arizona.
In 2019 and 2020, PA program faculty were nationally recognized as AAPA/PAEA Preceptors of the Year by the American Academy of Physician Assistants. The program was awarded for its Excellence Through Diversity in 2019 by the Physician Assistant Education Association.
Dedication to Data Analytics
Soon after the PA program accepted its inaugural class in Fall 2012, it began planning the implementation of a digital assessment platform with capabilities for reporting and analytics. The program recognized the importance of assessment data for benchmarking programmatic performance and preventing student attrition. They also knew faculty would need a tool to analyze assessment data by categories, including educational objectives, task areas, and accreditation criteria.
The PA program learned about ExamSoft’s item-tagging features, which would allow faculty to create an unlimited number of categories, organize categories into tiers, and assign multiple categories to individual exam items. With ExamSoft, the program would be able to break down performance by student, course, and program as well as run longitudinal reports to reveal category trends over time.
The PA program made the decision to adopt ExamSoft in Spring 2013 and began using the software to track performance by multiple categories, including Bloom’s Taxonomy of learning outcomes and educational objectives set forth by the Accreditation Review Commission on Education for the Physician Assistant (ARC-PA). When faculty create exams within the ExamSoft portal, they are able to tag new items to the appropriate category. Faculty can also assign more than one category to a single item, which allows the program to gather comprehensive performance data for every required outcome and objective.
We don’t just need a service to deliver an exam. We also need a service to analyze an exam with statistical measures…there’s just no other company that I have worked with that does the same amount of justice to statistical analysis as ExamSoft. — Dr. Elias Villarreal, PA Program Department Chair and Director
Building a Culture of Assessment
In the initial phase of the ExamSoft implementation, the PA program trained faculty to manage the software independently in their courses. Training new faculty to create, administer, and analyze assessments with ExamSoft has become an integral part of the onboarding process.
During the faculty hiring process, the PA program asks candidates whether they have any prior experience using ExamSoft. If a candidate is hired and has not previously used ExamSoft to administer assessment, the program will assign a faculty mentor to train them on the software.
The mentor will first perform a general walkthrough of the faculty portal to introduce basic platform features. Once the new hire is comfortable navigating the portal, the mentor will share best practices for question development in the program. New faculty write their exam items before the next meeting with their mentor, who will go through the steps of uploading items to the portal. With exam content successfully added to the item bank in ExamSoft, the new hire and mentor will create a mock exam — setting in-exam timers, inserting attachments, or enabling question and answer randomization as needed.
Once the new hire and faculty mentor have explored every aspect of exam creation, they will move on to discuss exam analysis. All core faculty in the PA program use ExamSoft to perform psychometric analysis of exam content; the faculty portal makes it easy enough to analyze these metrics without the help of a dedicated statistician.
Using metrics like the Kuder-Richardson Formula 20 (KR-20), faculty can determine the consistency of their overall exam, while measurements like point biserial and discrimination index can help faculty determine the quality of individual exam items. The new hire and mentor will review the analytics for their first exam and discuss departmental best practices for psychometrics. For instance, depending on the difference in performance between the upper and lower 27% of exam-takers, faculty may decide to change an item to a bonus question or throw it out altogether.
Once new hires are successfully onboarded, the PA program grants them access to the ExamSoft client support page, which features helpful how-to articles for navigating and troubleshooting the platform.
Dr. Elias Villarreal joined NAU’s PA program as Department Chair and Program Director in 2017. At his previous institution, Dr. Villarreal was an associate professor, academic coordinator, and main contact for ExamSoft training, onboarding, and faculty development. ExamSoft was already well-established in the PA program at NAU by the time Dr. Villarreal joined, and his extensive experience with the software has proved invaluable, particularly during student orientation at the Phoenix Biomedical Campus.
Each year, Dr. Villarreal develops a presentation for all matriculating students in the PA, PT, and OT programs. This presentation helps familiarize students with the main screens in Examplify —ExamSoft’s test-taking application — showing students what they will see when they log in, when there’s an exam ready to download, and what happens after a download.
At the end of the presentation, Dr. Villarreal shares information about a password-protected practice quiz with a variety of features enabled — including the highlighter, calculator, note pad, and spell checker — to help familiarize students with the in-exam experience. Each student is allowed 50 tries and six weeks to complete the practice quiz, which allows students to become familiar with Examplify before taking their first exam.
Streamlining Remediation and Retention
Performance Insights by Category
One of the main reasons for the PA program’s emphasis on category reporting in assessment is the motivation to help students succeed, both in the program and beyond. The program tracks performance in high-level categories like learning outcomes as well as practice-specific categories like organ systems or physical exam skills.
ExamSoft’s category reporting tools allow faculty to provide students with detailed feedback after each assessment. With Strengths & Opportunities Reports, faculty can generate individualized performance reports that show students the categories in which they are excelling and any areas where they need improvement. For instance, these reports may tell students that they are strong in cardiology, but weak in cardiac examination skills. The specificity of the feedback helps to more effectively direct study habits, which allows students to devote their time to improving the right skills.
Observing performance data at the course and program level is another way the PA program can identify students who may require additional support. Faculty will observe the data for standard deviations from the mean, which indicate low-performing students. After careful review of each student’s overall performance, faculty will decide if the student is in an acceptable range to continue the course or if the student is in need of remediation.
Tools to Prevent Attrition
For several years, the PA program’s attrition rate has held steady at one student per year. In nearly every case, student attrition is linked to a single course offered in year one — Human Anatomy. Because material in this course is naturally rigorous, the course offers unique remediation options, including the possibility of retaking assessments.
Throughout the semester, students can request meetings with faculty to review their Strengths & Opportunities Reports after completing each exam. If students fail any of the Human Anatomy exams, they may be given the opportunity to retake them at the end of the semester.
As an example of a best practice in one course, students need at least a 70 in the course in order to pass; if students score high enough on the first exam to bring their grade to a 70, they are not required to retake any of the remaining exams. As students retake each exam, their overall grade adjusts automatically within the ExamSoft portal, which allows faculty to quickly notify students once they’ve reached the required course grade.
Moving Forward with ExamSoft Solutions
When the COVID-19 pandemic prompted the pivot to remote learning in Spring 2020, the PA program implemented ExamID and ExamMonitor — ExamSoft’s auto-identity verification tool and A.I.-powered remote proctoring software — to deliver secure, remote assessments. A year has passed since then, and while the program is continuing to deliver remote exams, it anticipates the return of in-person assessments by Fall 2021.
Since the program recognizes that some students may not be able to return to campus as quickly as others, ExamID and ExamMonitor will allow faculty to take a hybrid approach to assessment — deliver secure exams both in-person and remotely. The program anticipates that these solutions will be useful for administering exams in interprofessional courses at NAU. Since the department has students at both the NAU’s Flagstaff campus and the Phoenix Biomedical Campus in Phoenix, ExamSoft’s remote testing solutions will allow faculty to deliver synchronous course exams for all students.
In Spring 2020, the program also implemented ExamNow — ExamSoft’s formative assessment tool – and ExamSCORE. But as faculty navigated the sudden transition to remote learning, many have not had the time to introduce these tools in their courses. Those who have been able to incorporate ExamNow use data from in-class, formative assessments to help students benchmark their performance in certain areas of the curriculum. With ExamSCORE, the goal is to create a process for OSCEs and rubrics for virtual case studies in conjunction with another platform. As the program prepares for the Fall semester, faculty anticipate having more time to dive deeper into solutions like ExamNow and ExamSCORE. Whatever future semesters have in store, NAU’s PA program plans to continue using ExamSoft’s reporting tools to make informed decisions in the program and provide students with valuable performance insights.