The College of Health Sciences at Alabama State University discovered that graduates who had performed well in the program were scoring lower than anticipated on board certification exams. After a careful review of the problem, they determined that paper-based assessments were to blame. By adopting ExamSoft’s digital assessment solution, the college was able to familiarize students with a computer-based testing environment, boost board exam scores, and expedite the grading process.
Background
Alabama State University in Montgomery was founded in 1867 as the Lincoln Normal School at Marion, the nation’s first state-sponsored liberal arts institution dedicated to the education of Black students. Despite facing a number of financial and political obstacles throughout its 154-year history, Alabama State has maintained its commitment to progress through education as envisioned by its founders.
Alabama State’s College of Health Sciences is one of the few interdisciplinary schools in the country with students from Physical Therapy (PT), Occupational Therapy (OT), and Prosthetics & Orthotics (P&O) programs participating in coursework together in their first year of training.
The PT and OT programs are among the few offered at historically black colleges and universities (HBCU), and the P&O program is the only one offered at an HBCU. Each of these programs trains general practitioners to serve diverse communities — particularly the medically underserved — with competence and compassion.
Challenge
Many students admitted to the College of Health Sciences live in rural areas of the state and have not experienced the same access to computers or reliable internet — much less computer-based testing — as students from metropolitan areas.
After looking at the national board exam scores among PT, OT, and P&O program graduates prior to 2018, the college noticed that some of its high-performing students who took exams delivered via paper and pencil weren’t scoring nearly as well on their board certification exams, which are administered in a digital format. The college knew that in order to boost board exam scores, it would need to phase out paper-based assessments and familiarize students with a digital testing environment.
Not only were paper-based exams hindering student success post-graduation, they were also negatively impacting faculty in the PT, OT, and P&O programs. Because of the interdisciplinary nature of many first-year courses in the College of Health Sciences, instructors of these courses typically administer more assessments in a single exam session than instructors in more focused courses. Scoring these large batches of paper assessments had become incredibly time-consuming, and it was difficult to observe course-wide trends in student performance and identify necessary improvements to exam items.
To increase faculty efficiency and better prepare students for successful careers in health care, the college decided that switching from paper-based assessment to a digital assessment platform was the best course of action. Chair of the PT and OT departments, Dr. Susan Denham, immediately began the search for a computer-based assessment software to create, administer, and grade exams in all three programs.
Susan first learned about ExamSoft while attending an education conference and thought the software would be a good fit. She proposed the idea of implementing ExamSoft to her colleagues, who agreed that it could be the key to solving the college’s challenges.
“A big reason for implementing ExamSoft was to get our students accustomed to a board-like scenario during testing.”*
— Dr. Susan Denham, Chair, Department of Physical Therapy and Department of Occupational Therapy
Adoption and Start Up
Once ExamSoft was approved for use in the College of Health Sciences, Susan and two of her colleagues became the first official users. The trio began using ExamSoft in a single course each, one from each program, to identify a set of best practices for faculty and students using the platform. One month in, Susan and her colleagues successfully delivered their first assessments, and with this test run complete, the college was prepared to onboard the remaining program faculty who opted to use the platform.
Susan, along with P&O Department Chair, Dr. Scott Bretl, recorded initial training sessions for faculty, and the college also offered in-services to field any questions as faculty began to manage their ExamSoft usage independently. Established instructors within the College of Health Sciences imported their existing exam content into the faculty portal, while some newer faculty used the portal to build entirely new exams.
Technology requirements at the College of Health Sciences ensured that most students had devices compatible with Examplify, ExamSoft’s test-taking application. The school hosted a mock exam for the PT, OT, and P&O programs, and faculty were pleased to find that students acclimated quickly to the exam environment.
Results
Following a near-seamless transition to computer-based assessment in the PT, OT, and P&O programs, the College of Health Sciences has made great strides in preparing students to pass board exams. With ExamSoft’s faculty portal, instructors have access to the question types and tools they need to create assessment items that more closely resemble board exam questions. For instance, since board exams are exclusively multiple choice, faculty choose this format most frequently for summative assessments.
Board exam scores among graduates of the PT program have increased by 18% since the switch to ExamSoft — accompanied by a curricular change — while board exam scores in the OT program have increased by 7%. In a 2020 of the P&O program, 100% of students reported that the program prepared them to be successful in residency positions.[1]
The ability to instantly score assessments within the faculty portal has transformed the grading process for instructors. Exams that took one anatomy instructor 30 to 40 hours to grade before implementing ExamSoft now take only three or four hours. Additionally, instructors can now leverage data via psychometric analysis to determine the efficacy of their exam items and improve the quality of their question banks for future exams.
“When we had to leave campus, ExamSoft worked better because of the different parameters that are built in. We can randomize questions and randomize answers and even have two different questions for the same exam. That enabled us to have exams that we felt were more cheat-proof.”
— Dr. Susan Denham, Chair, Department of Physical Therapy and Occupational Therapy
The Impact of COVID-19
Since the Spring 2020 pivot to remote instruction, the College of Health Sciences has trusted ExamSoft’s security tools and features to protect assessment integrity in the PT, OT, and P&O programs, in part through the support of an ExamSoft grant program. The college discovered that ExamSoft provided more security options than other remote testing platforms in use at Alabama State; unlike these solutions, settings within ExamSoft’s faculty portal allow instructors to enable question and answer randomization and set time limits for exam sessions. Instructors activated the complete device control feature to prevent all access to the internet and other computer programs during an exam session. This feature also disables screenshot capabilities to protect exam content.
Many students in the PT, OT, and P&O programs commute from rural areas of Alabama, and internet connectivity problems in those areas were magnified in the shift to remote learning. ExamSoft’s options for offline assessment have proven useful for students without reliable WiFi access. A reliable connection is not necessary for exams delivered in the offline mode — students only need internet access to download the exam in the days prior and upload the exam file once the exam is complete. This has allowed students to take exams at home, without interruption. Since the shift to remote learning and assessment, not a single student has been unable to take an exam as scheduled.
As of Spring 2021, instructors within the College of Health Sciences can choose between fully remote or on-campus exams in their courses. This hybrid approach gives faculty the flexibility to structure their assessments around the needs of their course. And in the event a student is unable to attend a scheduled exam on campus, faculty can easily duplicate and modify the exam for the student to access in a remote setting.
“Our long-term plan is to be able to provide students at the end of the semester or at the end of the academic year with a report that says, ‘Based on your exam results, you’re consistently testing in the 90th percentile for this standard, but in the 60th percentile for this standard.’ So students will have a little bit more of an idea of where their weaknesses are in relation to our standards, not having to just guess.”
— Dr. Susan Denham, Chair, Department of Physical Therapy and Occupational Therapy
Next Steps
Among the upheavals brought on by the pandemic, the College of Health Sciences has delayed its plan to use ExamSoft’s item tagging feature to link individual exam questions to accreditation standards in the PT, OT, and P&O programs. The goal of organizing items in this way is to make it easy for the college to generate reports for accrediting bodies and quickly demonstrate compliance. This plan is still on the horizon and will be the college’s primary focus once programs have fully adjusted to new processes.
Another important long-term goal for these programs is to dive deeper into Strength & Opportunities Reports in order to provide students with remediation more effectively. At the end of every semester or academic year, instructors will generate individual performance reports to help guide study habits as students prepare for the semester ahead. By using data from prior assessments, these reports can show students in what percentile they’re consistently testing for certain learning standards and measure their performance against the general performance of their cohort.
As new faculty join the PT, OT, and P&O programs and established faculty become more experienced with ExamSoft, creative ways of using the platform continue to emerge. This spirit of innovation drives programs within the College of Health Sciences to discover new methods for delivering and evaluating assessments, with the assurance that ExamSoft’s holistic assessment solution will have the tools necessary to bring their brightest ideas to bear.
[1] Prosthetics & Orthotics. Alabama State University. www.alasu.edu/chs/prosthetics-and-orthotics/prosthetics-orthotics
*This organization received consideration for participating in this Case Study.
