After two phases of curricular reform, the Dugoni School of Dentistry required a solution to align assessment content with professional competencies and reveal opportunities for improvement. The school paired ExamSoft’s item tagging and curriculum mapping tools to further unite teaching and testing, while strengthening content coverage in its curriculum.
Background
The Arthur A. Dugoni School of Dentistry at University of the Pacific in San Francisco, California offers six academic programs, including two predoctoral programs: the International Dental Studies (IDS) and Doctor of Dental Surgery (DDS). The IDS program is a two-year accelerated program that qualifies graduates to sit for any dental board in the country. The Dugoni School’s DDS program is the only program in the U.S. that offers an accelerated three-year curriculum.
The Dugoni School, through its “Head, Heart, Hands” philosophy, teaches an innovative, competency-based curriculum that emphasizes technical knowledge, communication skills, and a humanistic approach to patient care. Competency-based dental education requires students to demonstrate progressively complex behaviors and abilities in knowledge, skill, professionalism, critical thinking, and problem-solving — all toward preparing students for independent practice as dental professionals.
The Transition to Computer-Based Testing
As early as 2012, upon recommendation of a curriculum committee task force, the Dugoni School began its search for a computer-based testing software that could replace paper exams. The school selected ExamSoft’s digital assessment platform for its various advantages over paper-based testing, including enhanced security with complete device control, categorization of competencies via item tagging, and quick turnaround on performance reports to support student learning and inform teaching adjustments.
In planning for the implementation of exam item tagging, Dr. Cindy Lyon, Associate Dean of Oral Health Education, department chairpersons, and leading school faculty developed comprehensive curriculum content areas and competency statements. Once content areas and competency statements were established with group consensus, the school’s instructional designer, Dr. Sinky Zheng, developed processes for item tagging using ExamSoft categories and guidelines for longitudinal reporting. Academic Support Services then created item tags in ExamSoft’s faculty portal corresponding to each curriculum content area and competency statement. In 2019, the school made item tagging a requirement for faculty across all departments, which allowed faculty to drive assessment of curriculum content coverage over time, and the school to glean programmatic performance insights by category.
Strengths & Opportunities Reports
ExamSoft’s item-tagging system allows the Dugoni School’s faculty to generate individualized assessment reports that summarize student performance by each content area and competency. Provided faculty approval, students can also access these Strengths & Opportunities (S&O) Reports after exam scoring is complete through ExamSoft’s online student portal and see their performance on each exam question by category. If faculty have provided a “rationale” (written feedback for answers) when developing the exam question, then a student can also see why their answer is correct or not, turning assessment into a valuable learning experience.
In a 2018 study of digital assessment outcomes conducted at the Dugoni School, authors Dr. Zheng and Assistant Dean of Academic Affairs Dr. Daniel Bender cite positive feedback from students who received S&O Reports, which were found to be effective in focusing study efforts and promoting self-directed learning. Faculty have also benefited from student S&O Reports; a faculty focus group included in the study reported that S&O Reports helped to identify course-wide learning areas that required improvement and gave them the ability to make informed instructional adjustments.
Longitudinal Reports
Where Strengths & Opportunities Reports provide individualized assessment performance data, ExamSoft’s longitudinal reporting feature allows faculty and administrators to run reports showing students’ performance in each tagged content area and school competency over time. Longitudinal reports can be run by students themselves anytime, without faculty approval to access.
Report Findings: Computer-Based Testing at the Arthur A. Dugoni School of Dentistry*
- 90% of students felt comfortable testing with ExamSoft.
- 84% of students “agreed” or “strongly agreed” that ExamSoft is easy to use.
- 85% of students “agreed” or “strongly agreed” that S&O Reports are helpful for their learning.
* Zheng, M. and Bender, D. (2018). Evaluating Outcomes of Computer-Based Classroom Testing: Student Acceptance and Impact on Learning and Exam Performance. Medical Teacher. https://doi.org/10.1080/0142159X.2018.1441984.
Curricular Refinement and Mapping
Along with developments in category-based assessment and reporting, the Dugoni School has worked to refine its curriculum. In 2007, the school introduced a new integrated curriculum, the Pacific Dental Helix Curriculum, characterized by multidisciplinary learning and authentic assessment across departments and courses. The Dugoni School launched a second phase of curricular changes in 2017 to focus on students’ development of new skills, abilities, and values (competencies) required for the modern practice of general dentistry, and align curricular changes to the school’s “Head, Heart, Hands” philosophy centered around basic and clinical sciences, technical skills, and patient-centered care.
But the more successful the school has been at integrating and refining its curriculum, the more challenging it has become to organize the curriculum into a clear map of content areas and topics. And given the integrated design of the IDS and DDS programs, finding a user-friendly solution for curriculum mapping became a top priority for the school.
Goals for Curriculum Mapping
In October 2020, the Dugoni School participated in an ExamSoft grant program that offered the school a trial of Map, ExamSoft’s curriculum mapping tool. The school identified three goals that they hoped to achieve with Map:
1. Create quick and comprehensive curriculum maps
2. Organize the curriculum by required competencies and content areas
3. Handle future curriculum measurement integrations/additions without significant IT support
With Map’s intuitive, visual reporting, the school would be able view curriculum content areas and competency statement items in the IDS and DDS curricula to ensure proper coverage and refine each of these curriculum areas in assessment.
Curriculum Mapping Start Up
As the school prepared to launch Map, Academic Support Services created a template of all courses within the IDS and DDS programs. They then consulted with the school outcomes liaison team to create a data structure that could accommodate the school’s future curriculum and performance reporting needs. The school also hosted overview sessions to familiarize faculty leads with Map, and user trainings with Academic Support Services staff on using the software and running curricular reports.
Academic Support Services prepared a four-step plan to implement curriculum mapping with school curricula:
- Data Collection
• Faculty complete curriculum map worksheets for courses, selecting applicable competencies and content areas. - Data Entry
• Academic Support Services staff enter completed worksheets’ selections into ExamSoft Map. - Reporting
• Academic Support Services runs and exports curriculum map reports in software of all courses’ competency and content areas coverage. - Distribution
• Exported curriculum maps distributed to curriculum stakeholders for curriculum auditing, monitoring, or reviews.
• Map data is periodically audited and validated for continued accuracy.
Adapted from: Han, A.N., Laughter, L., Iyer, P. and Radif, M. (2022). A Competency-Based Online Curriculum Mapping Process for Dental and Dental Hygiene Education. Journal of Dental Education. https://doi.org/10.1002/jdd.12879
As a result of extensive planning and team collaboration, it only took about two months to get Map up and running for the IDS and DDS programs.
The nice thing about Map is that it’s very flexible. It is very adaptable to any data set, which helped us think creatively about how to put data in.
– Annie Han, Manager of Academic Support Services
Curricular Mapping Results
Because the Dugoni School takes a multidisciplinary approach to learning, it’s important for the school to identify any siloed courses or content areas and connect them to other parts of the curriculum. After implementing Map, the school’s first course of action was to look at the entire predoctoral curriculum across courses and departments and decide when and how to integrate IDS and DDS students.
With Map, the Dugoni School has been able to make several curricular improvements that have helped to further combine its courses and programs:
1. Horizontal Integration to Enrich Student Learning
Dr. Parvati Iyer, Course Director of Integrated Clinical Sciences, used Map to find out when and where her first-year class was learning certain subjects in other courses. With this information, Parvati has been able to deliver unified content to augment student learning and create points of connection across the curriculum.
2. Vertical Integration to Strengthen the Curriculum
Map can help faculty figure out which aspects of a certain topic students are learning, when, and in which course(s). Parvati figured out what occlusion-related topics year-two students were learning in other courses. With these insights, she was able to introduce additional occlusion topics in her year-one courses to build the foundational knowledge that students would need for occlusal analysis in year two.
3. Coverage Insights to Inform Curricular Additions
With Map, faculty can observe coverage by topic to see when and how thoroughly students are encountering these topics in the curriculum. These coverage insights help inform additions to the curriculum that enrich student learning. For instance, the school can reinforce learning in nutrition by adding more nuanced topics such as “nutritional deficiencies” and “oral manifestations of nutritional deficiencies.”
4. Fine-Tuning Assessment to Support Coverage Areas
Faculty can use Map to compare coverage in the curriculum to the content areas and competencies tagged to their exam items. This helps faculty fine-tune exam items if needed and ensure curriculum alignment with their course content coverage and their assessments.
Having Map in the ExamSoft portal is a great advantage, not only for use on faculty workstations, but also at home.
– Adrian Hernandez, Network Administrator, Information Technology
Next Steps with ExamSoft
Beyond supporting the Dugoni School in its efforts to integrate teaching and learning programmatically, the school anticipates that using Map will also help to improve the accreditation process by tailoring Map data reports to suit specific academic reporting purposes.
While the mapping tools that the school used in the past were effective for accreditation review, the school no longer needed these tools once the review process ended. Using Map will help to support curricular refinement throughout the academic year and provide the school with an evergreen solution to maintain compliance when the time comes for accreditation review. Documenting the curriculum throughout the year using Map reports will add to the school’s accreditation supporting evidence.
From an IT perspective, being able to access Map within a centralized, online portal is a huge improvement over past solutions, which required the IT team to set up physical access databases for every administrative system.
Moving forward, the Dugoni School will continue to develop its two-pronged approach of using category tagging along with Map to determine if the school is effectively aligning assessment coverage and student performance to mapped course content areas and competencies. By mirroring Map content areas to the same categories that faculty use for item-tagging, the school can ensure future alignment of teaching and testing while further integrating its programs to support its multidisciplinary curriculum and future generations of practice-ready professionals.
Curriculum mapping not only helps you design your courses to be successful, it also helps you understand the assessment piece that needs to match your teaching.
– Dr. Parvati Iyer, Course Director, Integrated Clinical Sciences I
This organization received consideration for participating in this Case Study.