When William Carey’s School of Pharmacy opened its doors in 2018, its faculty recognized the importance of data and reporting to programmatic success. The school adopted ExamSoft to deliver assessments for its accelerated Doctor of Pharmacy program, implementing category tagging to generate performance reports, provide targeted feedback, and streamline the accreditation process. The School received Full Accreditation Status in July 2021.
Background
Located in Biloxi, Mississippi, the School of Pharmacy at William Carey University offers an accelerated Doctor of Pharmacy program. The accelerated PharmD program is one of the few in the country and the only program of its kind in the southern Mississippi and Gulf Coast region. The school accepted its first cohort in July 2018 and received Full Accreditation from the Accreditation Council for Pharmacy Education (ACPE) in July 2021.
The Vision for WCU’s School of Pharmacy
Dr. Dave Weldon, Associate Dean of Pharmacy and Assessment, joined William Carey in 2017 to become one of the PharmD program’s founding members. A University of Mississippi alumnus, Dave learned about the soon-to-be School of Pharmacy from professional connections in the state.
Dave met with School of Pharmacy Dean, Dr. Michael Malloy, whose vision for the school resonated with his own — to provide an option for students who want to attend a Mississippi pharmacy school that is geographically more convenient to southern Mississippi and the Gulf Coast.
As an accelerated program, William Carey’s Doctor of Pharmacy provides an opportunity for both traditional and non-traditional students in the region to earn a professional degree in less time and without having to relocate.
Creating a Reliable Assessment Process
Shortly after Dave joined the School of Pharmacy as Associate Dean, the team began building a framework for assessment in the Doctor of Pharmacy program. The program would require a digital platform for summative assessment in the didactic curriculum. As the team began considering software options for the program, ExamSoft’s digital assessment platform was the clear leader.
With over six years of experience using the software at his previous institution, Dave knew that ExamSoft would enable the school’s Office of Assessment to gather the data it needed for comprehensive reporting. And the software’s approachable interface would make it easy for faculty to engage with every aspect of the assessment process. The school began the ExamSoft implementation in early 2018 as it prepared to receive its first cohort that summer.
Office of Assessment
Assessment Coordinator, Cynthia Grimes, joined Dave to form the Office of Assessment in February 2018. Given his extensive experience using the ExamSoft platform, Dave would serve as an administrative official for the assessment program in addition to his duties as Associate Dean.
The IT department helped get the Office of Assessment up and running with ExamSoft but soon turned over the reins to Cynthia and Dave as program administrators. The two would oversee management of the ExamSoft portal, creating user profiles for faculty and departmental administrators, enrolling students into the software, and training faculty to manage exams within the ExamSoft portal.
Faculty Training
Once the school began using ExamSoft to deliver summative exams in the PharmD program, the Office of Assessment planned to generate reports to track programmatic performance. But to obtain the granular data necessary to run these reports, faculty would first need to lay the groundwork by using category tagging in the ExamSoft portal.
Soon after core faculty entered the program, Cynthia and Dave demonstrated how to upload exam content to the faculty portal, assign programmatic categories to individual exam items, and use built-in analytics to determine exam and item quality. Once faculty were prepared to independently manage assessments in the portal, the Office of Assessment could begin using the data from summative exams to perform more advanced reporting functions.
I really had two main things that I wanted to accomplish with assessment — I wanted to gather fairly granular data, and I wanted it to be easy for the faculty to do that… And ExamSoft is really the only electronic testing software robust enough to do what we want it to do inside the Office of Assessment. – Dr. Dave Weldon, Associate Dean of Pharmacy and Assessment
Building a Foundation of Analytics
For Dave and his fellow School of Pharmacy founders, it was important to make reporting and analytics an integral part of the assessment process. The school decided to use programmatic categories to benchmark performance against educational outcomes and accreditation criteria, which would provide insight into the program’s strengths as well as opportunities for improvement. Faculty learned to connect their curricula to these programmatic objectives and make adjustments to exam content using psychometric reporting tools.
Outcomes and Competencies
The School of Pharmacy tracks programmatic performance in several categories: Educational Outcomes and Competencies (EOCs), Ability-Based Outcomes (ABOs), and Levels of Learning.
ABOs help fulfill each of the 10 EOCs, which align with the ACPE’s expectations for the PharmD program. The school’s Levels of Learning are derived from Bloom’s Taxonomy and distilled into three tiers from Bloom’s six.
After faculty upload their exams into the ExamSoft portal, they assign three tags to each item: ABO (which is grouped by EOC), Level of Learning, and course objective. With these tags in place, the Office of Assessment can generate programmatic reports that break down performance by category. These tags also allow faculty to observe trends in course-wide performance to help finetune teaching methods or restructure the course.
Psychometric Analysis
The Office of Assessment provided faculty with initial training for psychometric analysis in the faculty portal, but the departments in the PharmD program have their own best practices to decide what actions to take from assessment data. Using metrics like point biserial, as well as discrimination and difficulty indexes, faculty will decide if their exam items are the appropriate level of difficulty for the exam and the course.
Based on the results of the psychometric reports, faculty may decide to award bonus points for exam-takers who answered a question correctly. Conversely, if a large number of exam-takers missed a question that wasn’t intended to be difficult, faculty may decide to cover this material again in class.
Category tagging is absolutely critical; it’s not optional. It can’t be optional if you’re going to capture the data for your school. That’s the only way you can get a full picture of how you’re actually doing as a program. – Dr. Dave Weldon, Associate Dean of Pharmacy and Assessment
Pursuing the Path to Accreditation
Before the School of Pharmacy could accept its inaugural cohort in 2018, it had to apply for Precandidate Status with ACPE. To be eligible for pre-candidacy, School of Pharmacy administrators must show, to the accrediting body’s satisfaction, that they are qualified to lead the program and that the program has the resources necessary to offer a Doctor of Pharmacy.
The School of Pharmacy was granted Precandidate Status for the PharmD program in 2017. ACPE accreditors returned a year later to ensure that the program was prepared to enroll its first class — a process that includes reviewing the building blocks of the curriculum. ACPE determined that the PharmD program met all the criteria necessary to begin enrollment, and the School of Pharmacy was granted Candidate Status.
After the school achieved Candidate Status in 2018, the next step was Full Accreditation Status. Since Full Accreditation could not be granted until the PharmD program’s inaugural class graduated, the school turned its focus to gathering the data necessary to demonstrate ACPE compliance.
At the end of every program year, the Office of Assessment generates reports to demonstrate EOC coverage for ACPE review. As Assessment Coordinator, Cynthia condenses data from ExamSoft into a single-page summary of programmatic performance in each of the 10 EOCs. The simplicity of this educational outcome report makes it easy to visualize performance averages for each cohort and percentage increases from year to year.
By providing a birds-eye view of programmatic performance, these reports have helped to streamline accreditation visits and illustrate improvements from curricular adjustments. The School of Pharmacy’s first site visit for Full Accreditation occurred in March 2021. The initial results of the ACPE report showed full compliance in all areas, and the simplicity of program’s educational outcome reports was well-received by visiting accreditors.
ACPE granted Full Accreditation Status to the School of Pharmacy in July 2021.
William Carey School of Pharmacy
Educational Outcomes & Competencies (EOCs)
- Foundational Knowledge
- Critical and Innovative Thinking
- Ethics and Legal Decision-Making
- Systems Management
- Communication
- Pharmacy Practice and Provision
- Interprofessional Collaboration
- Promotion of Individual and Public Health Wellness
- Personal and Professional Development
- Commitment to Service
Gaining Actionable Data Insights
The benefits of tagging exam items to educational objectives extend beyond accreditation review. ExamSoft’s reporting tools have helped to improve student performance, while also providing insights necessary for curricular adjustments.
Strengths & Opportunities Reports
Every course in the program’s didactic curriculum releases Strengths & Opportunities Reports to students upon completion of an exam. Because faculty tag each of their exam items to programmatic categories and course objectives, exam-takers are able to see how well they performed in each of the areas based on their answers.
In addition to breaking down student performance by category, Strengths & Opportunities Reports allow faculty to show students the specific exam items they missed. Faculty have the option to release only incorrect answer choices; this allows them to give students more focused feedback to guide their study habits without compromising the integrity of exam content.
Course Objective Refinement
At the end of each term, the Office of Assessment collects exam data to run reports for courses and departments as well as the entire Doctor of Pharmacy program. Most recently, Cynthia has been organizing the course objective data for each course instructed as a tool for faculty to engage in a post-course review process.
Post-course reviews have also highlighted opportunities for course refinement within the School of Pharmacy. These reports help faculty compare objectives in their course curricula with programmatic categories. If an instructor determines that their course isn’t well-rounded in the program’s educational objectives, they are able to make adjustments to emphasize the underrepresented objectives.
By indicating the number of exam items that support a given course objective, post-course reviews help faculty decide if they are testing equitably in each objective. Occasionally these reports may prompt an instructor to reorganize their course objectives or combine several similar objectives into one broad objective. Revisions like these can provide a more complete view of course-wide performance and make Strengths & Opportunities Reports more effective for improving individual student outcomes.
Achieving Flexibility in Assessment
When the COVID-19 pandemic forced many colleges and universities to pivot to remote learning in March 2020, the School of Pharmacy was preparing to begin its third academic year. The school decided to start the Summer 2020 term with in-person instruction, holding classes in two separate buildings to ensure proper social distancing.
Delivering instruction and assessment on campus has been fairly seamless for the school, but some students have been unable to attend in-person sessions. To better accommodate exam-takers in both on-campus and remote environments, the school implemented ExamID and ExamMonitor, ExamSoft’s auto-identity verification tool and remote proctoring software.
By creating a duplicate exam within the portal and selecting an option to enable ExamID and ExamMonitor, faculty have been able to deliver secure, remote versions of the exams they deliver on campus. Any duplicate exam is automatically linked to the original exam within the portal, so there’s no need for faculty to combine assessment data.
Faculty can activate ExamSoft’s offline mode for both on-campus and remote assessments, to eliminate interruptions related to unreliable WiFi and to prevent instances of academic dishonesty. Complete device control blocks access to external programs, screensavers, and web browsers — everything but the test itself — while an exam is in session.
Having the flexibility that allows students to take assessments from any location is a tremendous benefit for the accelerated PharmD program. If students are unable to attend an in-person session for any reason, they can easily take the exam at home and eliminate the risk of falling behind in the program’s fast-paced curriculum.
Being in an accelerated program, exams happen frequently. If you wait four or five days to deliver a make-up exam, there might already be another exam happening, and students can easily fall behind. – Dr. Dave Weldon, Associate Dean of Pharmacy and Assessment
Looking Ahead with ExamSoft Solutions
The School of Pharmacy strives to gather data and generate reports that provide a complete view of performance in the PharmD program, and delivering every assessment with ExamSoft would make this possible. While all summative assessments are currently administered using the software, there are clinical activities in the first two years of the program scored outside of ExamSoft.
The school sees the potential for using ExamSCORE to assess clinical activities, including OSCEs. This rubrics-based tool would allow instructors to score students in a clinical setting, tag rubric areas to programmatic categories, and collect performance data for reporting in the Office of Assessment.
The William Carey School of Pharmacy’s commitment to assessment data and reporting has contributed to the success of the Doctor of Pharmacy program, from achievements in accreditation to category-driven performance insights. As the school continues to use data to drive decision-making, it further supports its mission to prepare students from Mississippi and surrounding areas for successful careers in pharmacy.
