In 2014, the College of Pharmacy & Health Sciences at Sullivan University decided to optimize testing in its PharmD program. After making the switch from paper-based exams to digital assessment with ExamSoft, the school has reaped the benefits of the assessment software’s data and reporting, including psychometric analysis and category tagging.
Founded in 1962, Sullivan University is a private institution with locations in Ft. Knox, Lexington, and Louisville, Kentucky. The College of Pharmacy & Health Sciences in Louisville offers several programs, including the Doctor of Pharmacy (PharmD) and Master of Science in Physician Assistant (MSPA) programs.
In 2008, Sullivan accepted its first class of students into the PharmD program, a three-year, accelerated program In the summer of 2014, Sullivan welcomed its inaugural class into the MSPA program, a two-year curriculum split into didactic (one year) and clinical phases (one year).
In early 2014, PharmD program faculty decided that their current assessment practices were unsustainable. At the time, they were administering paper-based and bubble-sheet assessments to nearly 100 students across all courses. The process of printing and distributing exams was time-consuming, and faculty weren’t directly involved in the process of scoring bubble-sheet exams. With faculty creating quizzes and exams using a word-processing program, there was no way to observe performance trends across students or analyze the efficacy of individual items.
Dr. Sarah Raake, Associate Professor and Director of Institutional Effectiveness, was a member of the Information and Technology Advancement Committee (ITAC) when the program leaders decided to update their assessment practices. The committee was responsible for exploring edtech to use in the College of Pharmacy & Health Sciences. Sarah and her ITAC colleagues began to research options for digital assessment. The ideal assessment solution would help faculty boost exam efficiency and give them more comprehensive exam data. Another top priority was finding a platform with capabilities for offline testing to allow testing without the need for internet connectivity.
ITAC selected ExamSoft’s digital assessment platform, particularly for its offline testing options, and faculty in the PharmD program would soon explore the potential of the software’s data and reporting capabilities.
Adoption & Start Up
Over the next couple of quarters, the Office of Academic Affairs and Assessment (OAcA), of which Sarah is a part, selected a handful of courses to pilot ExamSoft in preparation for a full rollout in the PharmD program. The faculty of these courses had varying levels of proficiency with education technology, and these first encounters with the software would help the OAcA develop effective job aids, tutorials, and trainings to benefit all faculty moving forward, no matter their experience with testing software.
Within just a few quarters, the OAcA had prepared the resources necessary to launch ExamSoft throughout the PharmD curriculum. For six months after the launch, the OAcA offered full-day and half-day ExamSoft training sessions to help ease the transition. Each session covered specific aspects of the software, from uploading exam items to understanding psychometric reporting. The OAcA also created a designated support team to assist with any software questions or issues and directed faculty to helpful how-to articles from the ExamSoft Community page.
When the PharmD program first began using ExamSoft, a fair number of its students had never tested on a computer. The OAcA scheduled mock exam sessions to help students gain experience with the software before taking graded tests. Since students were already facing the academic rigors of the PharmD program, the OAcA used these practice sessions to alleviate any exam-day stress.
When Ben Stephens, M.Ed., joined the College of Pharmacy & Health Sciences as an instructional support specialist in 2017, the PharmD program was using ExamSoft to deliver every formative and summative assessment. Faculty in MSPA program were beginning to use the software as well. Helping faculty to leverage ExamSoft most effectively in their courses was soon to become a large part of Ben’s role, and he now serves as the key ExamSoft administrator for the school.
Over the years, Ben and Sarah have worked together within the OAcA to expand ExamSoft usage in the program — developing faculty training programs and tools as necessary and laying the groundwork for effective reporting.
Category Tagging & Programmatic Reporting
As faculty and students in the PharmD and Physician Assistant programs have become more accustomed to using ExamSoft, the OAcA has pushed the program to explore new aspects of the software, particularly its data and reporting capabilities. The OAcA focused on ExamSoft’s category-tagging feature, with the goal of revealing performance trends and content coverage at the course and program level. Generating these category-based performance reports required instructing faculty to assign categories to individual exam questions.
The OAcA collaborated with the Associate Dean of Academic Affairs and Assessment to identify five core categories to track within the PharmD and Physician Assistant programs depending on program needs. In the PharmD program, for example, mapping includes: faculty name, Bloom’s Taxonomy level, Center for the Advancement of Pharmacy Education (CAPE) outcome, Pharmacy Curriculum Outcomes Assessment (PCOA) standard, and content area. With faculty tagging their assessment items to these five categories, the OAcA can run longitudinal performance reports to demonstrate that the PharmD and Physician Assistant programs are meeting the required criteria for their programs.
Triple-Check Report for Category Tagging
Instructional Support Specialist, Ben Stephens, M.Ed., developed a method to ensure consistent category tagging across the PharmD and Physician Assistant programs. Every assessment item in the program includes tags that show how the item satisfies criteria within each of the program’s five core categories. These tags indicate the category subgroup, with the main category in brackets.
Ben created a spreadsheet that allows faculty to confirm that their exam items are tagged to each of the programmatic categories. Faculty can copy assessment data from ExamSoft, including question title and categories, into the spreadsheet. The spreadsheet then analyzes the category tags to make sure each of the main categories are included. If any category tags are missing, faculty can easily locate the item within ExamSoft and make the adjustment.
Sample Entry from the PharmD Triple-Check Report:
|Question Title||Categories||Category 1: Faculty Name||Category 2: Bloom’s Level||Category 3: CAPE Outcome||Category 4: PCOA Standard||Category 5: Content Area|
|Pharmacy Question 1||Raake, Sarah [Faculty]|
1.1: Learner [CAPE]
|Pharmacy Question 2||Raake, Sarah [Faculty]|
Pharmaceutical Sciences [PCOA]
Post-Exam Reports for Retention & Remediation
Tagging exam items to categories also enables faculty to generate ExamSoft’s Strengths & Opportunities (S&O) Reports — individualized, post-exam reports that break down students’ exam results by category to reveal areas where they performed well and areas where they can improve. While not all PharmD and Physician Assistant faculty currently use S&O Reports, these reports play an important role for Sarah in her student remediation efforts. Sarah schedules one-on-one meetings with students who need extra support, prints their S&O Report, and uses results from their report to guide the session. She customizes the reports to show performance in certain categories, as well as correctly answered questions, incorrectly answered questions, and the rationale for each question. With these insights, Sarah can show students which content areas they need to revisit as they prepare for future exams.
Using S&O Reports for PANCE Success in the MSPA Program
Faculty in the MSPA program are using S&O Reports to prepare students for the Physician Assistant National Certifying Exam (PANCE). Every year, the program delivers an end-of-curriculum exam with items tagged to PANCE-related topics and content areas. After students complete the exam, faculty release S&O Reports that show them how they performed in each category — without revealing specific questions or answers on the exam. By customizing S&O Reports in this way, faculty can give students the valuable feedback they need to pass the PANCE while maintaining the integrity of exam content.
With the help of psychometrics, exam administrators can determine the effectiveness of individual items and entire assessments. Before adopting ExamSoft, faculty in the PharmD and Physician Assistant programs had limited experience using psychometrics to refine their exams or question banks. Dating back to the initial ExamSoft training sessions, the OAcA has gradually worked to educate faculty on psychometrics, demonstrating how to use the built-in reports from ExamSoft.
As more PharmD and Physician Assistant faculty began to use psychometrics to adjust exam items, the OAcA noticed varying interpretations of psychometric results. To help guide more consistent decision-making in the program, Ben developed an Item Adjustment Support Workbook for faculty use. Faculty paste the psychometric results that they export from ExamSoft into the workbook, which then suggests an approach based on the scores using embedded IF functions.
For instance, if an item has a very low value for point-biserial, the spreadsheet generates a response that suggests awarding full credit to students who missed the item. If an item has a low value for difficulty index and a high point-biserial, the spreadsheet indicates the item may be better used as a bonus question. This workbook gives faculty the assurance that they are interpreting their psychometric data correctly and making reasonable item adjustments, all while making exam grading more consistent from course to course. [Download the full case study to see an example.]
Switching to ExamSoft has helped to promote NAPLEX- and PANCE-readiness for students in the PharmD and Physician Assistant programs. Using the digital assessment platform for all formative and summative assessments prepares students for the experience of taking high-stakes, computer-based exams. When taking the NAPLEX and PANCE, students cannot return to previous questions to adjust their answers. With ExamSoft, faculty in the PharmD and Physician Assistant programs can disable backward navigation for their exams to familiarize students with this style of testing.
To prepare students for the kinds of items that appear on the NAPLEX, PharmD faculty in the program are working to incorporate more hotspot questions, which prompt students to select an area or multiple areas of an image in response to a question. ExamSoft offers a variety of item types, including hotspot items, so faculty can build assessments that go beyond traditional multiple choice to better replicate high-stakes exams.
The PharmD program is expecting a site visit from the Accreditation Council for Pharmacy Education (ACPE) in 2024, and the OAcA plans to use ExamSoft’s data and reporting tools to streamline the reaccreditation process.
Before ExamSoft, the program used spreadsheets that they created by hand, along with anecdotal evidence, to demonstrate compliance during site visits. Preparing for these visits could be time-consuming, since finding the necessary documentation meant digging through paper files and assessments stored in more than one place.
The program hadn’t fully explored ExamSoft’s assessment tools in time for its last accreditation review. This time around, the OAcA will use the software to demonstrate where and how the PharmD curriculum is meeting accreditation standards and highlight continuous improvement — in individual courses and throughout the program.
With ExamSoft, program administrators can download the reports required to demonstrate accreditation compliance in a matter of seconds. When program admins want to create customized reports, the software exports assessment data into well-organized, user-friendly spreadsheets.
ExamSoft Assessment Conference 2022
Sarah and Ben are both planning presentations for the 2022 ExamSoft Assessment Conference (EAC). In “Enforcing Category Tagging: Balancing Academic Freedom with Consistency,” Sarah and Ben will explain how they streamlined category tagging in the College of Pharmacy & Health Sciences, including the process of developing the Triple-Check Report. Sarah will also be contributing to “Exam Review: Augmenting Student Learning Even After the Exam is Submitted,” which will explore options to conduct secure exam reviews that preserve assessment integrity while also supporting student learning.