Since the D’Youville School of Pharmacy opened its doors in 2010, it has recognized the importance of assessment data for achieving programmatic excellence. As the school’s assessment process evolved, faculty in the Doctor of Pharmacy program required more comprehensive reporting than its solution at the time would allow. In 2015, the school adopted ExamSoft for its extensive item-tagging capabilities and data-rich performance reports.
Background
Located in Buffalo, New York, D’Youville College was founded in 1908 as the first school in the region to offer baccalaureate degree programs for women. Beginning with an inaugural class of nine, D’Youville has grown to offer 56 programs to a coeducational student body of 3,000 graduates and undergraduates from diverse backgrounds.
The D’Youville School of Pharmacy accepted its first cohort in 2010 and today offers a Bachelor of Pharmaceutical Science (BSPS), a Doctor of Pharmacy (PharmD), and a Pre-Pharmacy (Early Assurance) program. A major focus for the School of Pharmacy Dean, Dr. Canio Marasco, is keeping the school on the leading edge of the pharmacy profession and preparing graduates who are practice-ready today and five years down the road.
With a curricular emphasis on service learning, the School of Pharmacy imbues students with a strong commitment to improving the lives of patients in their community. D’Youville is soon to open a Health Professions Hub — in partnership with Catholic Health Buffalo — offering clinical services to underserved populations of Western New York. Once the Hub is approved as a point of distribution for COVID-19 vaccination, the School of Pharmacy will serve the vital role of administering vaccines to all eligible patients.
Challenge
In its first years, the School of Pharmacy was using a software solution to organize assessments and assignments by each of its accreditation competencies and educational objectives. But as the burgeoning school became more established in its practices, faculty in the PharmD program began to notice inefficiencies with the software.
The software did not offer a convenient option for generating aggregate data reports, which are essential for measuring programmatic performance and proving compliance to its accrediting body. In order to demonstrate how their objectives and standards were linked to performance in the PharmD program, the Director of Assessment had to gather individual examples of student work and provide a written summary for each, explaining their relevance to a given outcome. Not only was this process time-consuming, but it also lacked the robust, consolidated insights that a comprehensive digital reporting tool could provide.
Determined to find a more intuitive solution for categorizing assessment data and observing performance trends, the school began its search for new software. School administrators considered several options but were particularly drawn to ExamSoft for its item-tagging capabilities for assessment, which are designed to facilitate curricular alignment with educational objectives and accreditation standards.
In 2015, the School of Pharmacy made the decision to adopt ExamSoft for delivering summative assessments in years 1-3 of the PharmD program.
Adoption & Start Up
The ExamSoft rollout coincided with curricular changes in the School of Pharmacy, which included the transition to a new set of learning outcomes. Director of Assessment and Accreditation, Dr. Amany K. Hassan, was Chair of Curriculum at the time of implementation and helped decide which outcomes the school would be tracking in the ExamSoft faculty portal. Dr. Hassan remembers the extensive training efforts on the part of the assessment team at the time, who developed a gradual plan for introducing faculty and students to each aspect of the ExamSoft platform.
Once the school finalized its new outcomes, the assessment team began training faculty to upload items to the portal and create their own exams. The second phase of training covered exam-reporting capabilities — teaching faculty how to measure the quality of exam items with psychometrics. Once faculty were comfortable with creating assessments and analyzing exam data, the assessment team instructed faculty to tag individual exam items to major educational outcomes and competencies in the PharmD program. One year later, faculty began tagging items to the subcategories of these outcomes and competencies.
Once faculty were up and running with the ExamSoft portal, the School of Pharmacy began to transition students to the assessment platform. PharmD students are expected to supply their own computers as a requirement of the program, so students were able to download Examplify — the student test-taking application — to their personal devices. The school provided backup laptops in case a student needed one at the last minute, and the assessment team distributed a list of best practices to help troubleshoot any issues students may encounter on exam day.
After a deliberate, two-year implementation process, all faculty were prepared to independently manage their ExamSoft usage, and students had the resources they needed to ensure a smooth exam session. In 2017, all of the School of Pharmacy’s assessments were successfully administered with ExamSoft.
Everybody was responsible for creating their own exams, and that is why we needed the training — we needed faculty to understand the process. We also had trainings to produce reports showing progression and achievement of educational outcomes and for faculty to know, ‘What does point-biserial mean?’, ‘How do you measure if this is a good question or a not-so good question?’*
– Dr. Amany K. Hassan, Director of Assessment and Accreditation, Associate Professor
Results
ExamSoft’s item-tagging and reporting capabilities have become integral to the School of Pharmacy’s educational assessment of its PharmD program. As of 2020, all summative assessments in the program are administered via ExamSoft, and at least 95% of projects and presentations are scored using ExamSoft.
Data from these high-stakes assessments and assignments has allowed the school to demonstrate what it is assessing during its annual review of student learning outcomes, Bloom’s Taxonomy learning objectives, and standards set forth by the Accreditation Council for Pharmacy Education (ACPE). Academic assessment of the PharmD program helps the School of Pharmacy maintain programmatic compliance with accreditation criteria and ensure its graduates are practice ready.
Every semester, students in the PharmD program are required to complete e-portfolios demonstrating that they have met service requirements, core curricular skills, and educational outcomes. Included in this portfolio are longitudinal reports from ExamSoft, which organize student performance by each of the program’s educational outcomes. These reports — paired with other components of the e-portfolio — help the program determine that PharmD students have met all the necessary program requirements of the didactic curriculum before entering the experiential curriculum in year 4.
In 2020, the School of Pharmacy adopted Map — ExamSoft’s curriculum mapping tool — to determine the frequency with which it is covering foundational knowledge in the PharmD program. Faculty tag course syllabi to Appendix B assessment objectives provided by ACPE. Using Map’s “Coverage View,” the school is able see the number of times a given topic is covered across the curriculum and determine if that is an adequate number for supporting student mastery.
Before adding Map, the school used Microsoft Access and Excel to organize its curriculum by topic. But using an open-source spreadsheet for adding syllabi made it difficult to keep data organized during faculty collaboration. Map serves as a centralized communication hub for faculty members and administrators and organizes data into a clear visual representation of the PharmD curriculum, giving stakeholders a reliable tool for making curricular decisions.
The School of Pharmacy’s extensive use of item-tagging and category reporting has helped further its goals toward programmatic excellence. The PharmD program is fully accredited by the ACPE, and as of 2019, PharmD graduates have achieved a nearly 85% pass rate for the North American Pharmacist Licensure Examination (NAPLEX) and report a 96% job placement rate within six months of graduation.
We went from delivering paper-based exams to digital exams via ExamSoft. And really, considering all that’s happened this past year, it was quite a fortuitous blessing because we were able to — within a matter of five days — transition to online assessment. So, we were able to deliver exams and quizzes remotely in a secure manner.*
– Dr. Canio Marasco, Dean and Professor, D’Youville School of Pharmacy
The Impact of COVID-19
At the outset of a worsening pandemic, colleges and universities were forced to tackle a mid-semester pivot to online learning, and programs at D’Youville were scrambling to transition their classrooms, quizzes, and final exams to a remote setting.
When COVID-19 hit, the School of Pharmacy had been using ExamSoft to administer digital assessments in its PharmD program for three years. Measuring educational objectives and accreditation standards via assessment data is central to the school’s practices, so protecting the integrity of that data in a remote setting is essential.
Faculty in the PharmD program activated Examplify’s offline assessment mode to deliver secure exams at home. Examplify uses complete device control in the offline mode, which ensures that students cannot access any other programs on their devices while an exam is in session. Using the offline mode for remote exams has helped maintain the school’s confidence in its assessment data. Any opportunities for academic dishonesty could compromise data for the entire academic year, which could in turn affect the school’s accreditation status. Secure, offline assessment with ExamSoft helped the School of Pharmacy stay on track with its programmatic goals.
Moving Forward
One of the legacies of the School of Pharmacy’s founding dean, Dr. Gary P. Stoehr, is its culture of assessment, and for the past five years, ExamSoft has played an important role this culture. The school recognizes the importance of assessment data, helping faculty develop an impressive level of expertise with ExamSoft and make data-driven decisions to inform their teaching.
Despite many unknowns surrounding the return of in-person learning, the school has the flexibility it needs to protect assessment integrity — whether online or on campus — with ExamSoft. Moving forward, the School of Pharmacy is focused on continuous quality improvement in the PharmD program and finding ways to further optimize the assessment process. ExamSoft will continue to provide valuable insights to help the school identify areas of improvement for programmatic excellence and educate future-minded professionals in the field of pharmacy.
*This organization received consideration for participating in this Case Study.
