In early 2020, RESP-FIT began its search for a digital assessment solution to serve candidates across Australia seeking accreditation for respirator fit testing. The program selected ExamSoft to centralize management of its theoretical and practical assessments, preserve exam integrity with technology-enabled security features, and gain exam insights to drive continuous improvement.
Background
RESP-FIT is a respirator fit testing training and accreditation program, developed and managed by the Australian Institute of Occupational Hygienists (AIOH). The program promotes correct use of respiratory protection by confirming whether the fit of a certain respirator achieves an acceptable fit for the wearer, in accordance with necessary safety standards and protocols.
RESP-FIT’s nationally recognized accreditation process is intended for individuals in Australia to demonstrate competence in respirator fit testing. RESP-FIT offers accreditation in three fit test methodologies; to obtain accreditation in a specific methodology, exam-takers must complete a two-stage assessment process, which includes a multiple- choice, theoretical exam and a practical assessment video.
Challenge
As RESP-FIT prepared to launch its accreditation program in December 2020, Program Chair, Mark Reggers, along with several key members of the RESP-FIT board, began researching methods to deliver its theoretical and practical exams.
As a small, volunteer-led program serving professionals across Australia, RESP-FIT required a solution that could support the remote collaboration of assessors within the program and accommodate exam-takers testing from multiple locations throughout the country.
The program decided to pursue a digital assessment solution with capabilities for remote testing, which would provide a centralized portal to create, deliver, and score exams and eliminate the need for exam-takers to travel to an in-person testing center.
Adoption
After months researching solutions for online assessment, RESP-FIT selected ExamSoft in mid-2020. The platform would enable RESP-FIT to administer exams to candidates in various locations across the country. Being able to test remotely was important for RESP-FIT to ensure
the timely launch of its accreditation program, especially since the COVID-19 pandemic had forced many testing organizations around the world to suspend in-person exams.
ExamSoft’s security features help RESP-FIT uphold exam integrity in a remote setting so its assessors could be sure that exam-takers had truly met the criteria necessary to become accredited. Using complete device control, assessors disable access to the internet and any external programs on exam-takers’ devices while the exam is in session. ExamID and ExamMonitor — ExamSoft’s auto- identity verification tool and technology-powered remote proctoring software — work together to digitally verify exam-takers’ identity and detect any instances of academic dishonesty throughout the exam.
Another key feature that drew RESP-FIT to ExamSoft was the rubrics-based grading solution, ExamSCORE. With this tool, assessors at RESP-FIT can evaluate practical video submissions using digital rubrics, centrally stored in the ExamSoft portal along with results and analytics from theoretical exams. The ability to manage both exams required for accreditation in the same platform set ExamSoft apart from other exam solutions.
“Our decision to adopt an online-first approach to assessment was further amplified and validated with the pandemic. We would not have been able to launch our accreditation process without ExamSoft.”
— Mark Reggers, RESP-FIT Program Chair
Start Up
Once RESP-FIT made the decision to adopt ExamSoft, the assessment team collaborated to create a pool of exam items and rubric areas that evaluated each of the theoretical and practical skills required for accreditation. The team also identified the set of international fit testing standards and program-specific objectives to connect to each exam item using ExamSoft’s category tagging feature.
RESP-FIT rolled out ExamSoft in stages between October and December 2020 — first to a pilot group of internal assessors and then to a select group of exam-takers before launching the official exam. The program created a mock exam through ExamSoft as an initial step of its assessment process. This allows candidates who register for the RESP-FIT accreditation to become familiar with the platform before taking the theoretical portion of the exam, and it also prompts candidates to set a baseline photo that ExamID then uses to verify their identity on exam day.
RESP-FIT makes the exam available a week before exam day, which gives candidates ample time to download the exam file to their devices to also check system and security device compatibility. Once exam day arrives and at the nominated time, the program distributes passwords that exam-takers use to access the theoretical exam. Upon completion of the exam and review of the ExamMonitor file, applicants are provided a Strengths & Opportunities report to provide feedback on element performance. For those required to resit the exam this assists with exam preparation.
The practical assessment portion is the second phase of RESP-FIT accreditation. A group of around 25 assessors review the practical video submissions and score them using rubrics in ExamSCORE. The rubric areas measure exam- takers’ ability to set up fit test equipment, conduct a fit test, identify causes of a fit test failure and other key criteria. To help support the reliability and validity of exam results, RESP-FIT requires each assessor to declare any potential conflicts of interest for the candidates who submit their practical assessments.
Results
ExamSoft’s digital assessment solution has allowed RESP-FIT to streamline the accreditation process while also accommodating candidates located throughout Australia. Though remote testing is not yet widespread in similar certification programs, RESP-FIT has successfully partnered with ExamSoft to facilitate a secure and reliable exam experience, consistent with that of in-person testing. As of June 2022, RESP-FIT has delivered accreditation exams to over 200 candidates.
RESP-FIT Theoretical Exam
The theoretical exam is a timed, multiple-choice assessment that covers theory for basic respiratory protection selection and respiratory protection programs. Refining the theoretical exam was a top priority for RESP-FIT in the first six months after launching ExamSoft, so the assessment team relied on the software’s psychometric data to analyze the overall efficacy of its exam and revise items as necessary based on their findings. ExamSoft’s digital question bank makes it easy to adjust individual items and track these adjustments over time, so RESP-FIT can keep a record of item revisions for easy reference.
After delivering the first few exams, the assessment team noticed certain trends in the performance reports that could improve the process for marking exams. The team discovered several exam items that were basic indicators of whether candidates were competent to become accredited. They decided to adjust the weight of these items within ExamSoft so they were worth more points than other items included on the exam. With this adjustment, exam-takers in order to pass the theoretical exam they need to answer every one of these weighted questions correctly to be more likely to pass the theoretical exam.
RESP-FIT continues to analyze the results of each exam it delivers, reviewing the performance reports to determine if any items should be marked as bonus questions or whether to make any revisions for future exams.
RESP-FIT Practical Exam
Once candidates complete the theoretical exam, they have three months to upload the link to their practical assessment video. In the video submissions, candidates perform each step of the fit test methodology for which they’re seeking accreditation. Assessors at RESP-FIT then review each submission and evaluate candidates’ performance using rubrics through ExamSCORE.
The assessment team has created both mandatory and supplementary rubric areas for each of the methodologies they assess, which helps them quickly identify exam-takers who meet the basic criteria for accreditation. If they score 80% or higher on the practical exam but do not meet all the mandatory rubric criteria, RESP-FIT releases ExamSoft’s Strengths & Opportunities (S&O) Reports to each of these exam-takers that show the specific categories where they need to demonstrate improvement. The program then allows them to submit a supplementary video that covers the area(s) of improvement noted in their individual S&O Reports. From there, assessors review the supplemental video and adjust scores accordingly within ExamSCORE.
Remote Invigilation
Because RESP-FIT delivers remote assessments, having a method to protect exam integrity is important to preserving the value of its credential. The program uses ExamID to automatically verify the identity of candidates before the exam, along with the technology-enabled proctoring tool, ExamMonitor, which observes exam sessions via audio and video to detect potential security breaches.
Reviewing suspected breaches is more efficient with ExamMonitor, which only flags recordings with unusual behavior and even highlights specific instances in the recording where a security breach may have occurred. This allows assessors to quickly determine if there was a breach rather than reviewing all the recordings for a given exam session.
“Our accreditation exam is a professional undertaking protecting worker health, and the invigilation process with ExamMonitor helped to maintain that level of professionalism.”
— Mark Reggers, RESP-FIT Program Chair
Next Steps
Having successfully laid the groundwork for remote assessment with ExamSoft’s solutions, RESP-FIT is well-positioned to expand its accreditation process to international applicants, in addition to the candidates it already serves across Australia. Having a scalable solution for assessment is particularly important for the small, volunteer-led team at RESP-FIT, and ExamSoft provides a stable infrastructure to accommodate a growing pool of candidates. RESP-FIT’s exams are completely digital, so there’s no concern over costs to print exams or book testing venues.
As more certification programs explore the potential of remote assessment, RESP-FIT’s accreditation process — and all that the program has partnered with ExamSoft to achieve — serves as a model of success in the domain of assessing respirator fit tester competence.
