AIRS Certification Program Computer-Based Testing (CBT)
AIRS Webinar: October 2013
The Original Way • AIRS Certification has been around for over 15 years • Eligible candidates take a 100 question exam and acquire either CIRS, CIRS-A or CRS Certification if they pass the exam • The exams have always been proctored with testing taking place in classroom style settings
The Original Way • And the exam answers have always been entered and marked using Scantron forms
Problems with the Original Way • Scantron forms are not the friendliest pieces of paper. It is a strain on the eyes and requires a reasonably intense focus • Although machine readable, AIRS Certification staff often hand-scored exams if the score was within 5 of a passing mark or it looked like a candidate had made an honest mistake in alignment • The exam questions themselves were always at risk as boxes of question papers were sent around the US and Canada, then returned
Problems with the Original Way • Providing timely results to candidates was also a constant challenge – we had to wait for the boxes to be returned, retrieve the forms, feed them through the machine, verify the results – and during busy times, there would be boxes going in and out every day, and if one person was off for a couple of days, then the backlog would build up, etc. etc. … • Candidates wanted their results within a few days. Our target was 30 days and sometimes, we were not even able to meet that …
First Look at Computer-Based Testing • CBT has been on the AIRS radar for about 6 years • Original concerns were: – Expense to members (costs of up to $100 per person) – Lack of location choices (most operated through commercial testing centers – less options, even more expense) – Concern of impact on Affiliate conferences – Sense that not everyone was ready for the switch – AIRS Certification had other priority (from 2008 to 2012, all AIRS Certifications underwent stringent psychometric review with new questions and new exams)
Second Look at Computer-Based Testing • More software options with lower pricing and more flexible options • More unease about Scantron as suitable for persons with visual impairments • Survey of AIRS members in 2012 showed that 40% of members had a training room with computers/Internet
Second Look at Computer-Based Testing • Do you know of a 'training-type room' within your community (for example, an employment center, a library, community college) that has a number of Internet-capable computers that could be easily accessed for a CBT Certification exam? • Yes, and I am aware that either my own or other agencies have used this space for various reasons = 39% • Yes, I am aware of such a place but I have never had cause to contact them = 33% • No, but I imagine such a space exists and obtaining it should not be an obstacle = 15% • No, this may be a problem = 13%
Second Look at Computer-Based Testing • Which of the following best describes your feelings towards Computer-based testing? • I fully welcome it = 69% • I welcome it but have some concerns= 27% • I am concerned about it but appreciate there are some benefits= 10% • I prefer the existing system = 4%
Conclusion: Let’s Make the Move to Computer-Based Testing
Software Selection • After testing several CBT softwares, AIRS chose an application called Questionmark • What we liked about it: – Admin end of it was relatively easy to use – Existing clients included US military, major technology companies, financial services, government – Price … honestly, it was the only one that would not have driven up the cost of Certification for AIRS members
Software Selection • Over past 12 months: – Loading of 800+ questions each with 4 option choices – Testing of software by AIRS Certification Commission members, volunteer agencies, Affiliates – Framing and refining of documentation – More testing – Bit more testing
Operating for past few months on Stealth Mode
But now CBT is the only option for AIRS Certification exams …
BUT THERE ARE OPTIONS WITHIN THAT OPTION …
CBT: What stays the same • This is not a low stakes Certification exam. There is still a requirement for a proctor • The application process to be approved as a candidate to take an AIRS Certification exam remains the same • AIRS Affiliates still need to approve tests taking place in their state/regions • The ‘atmosphere’ and ‘general expectations’ of an exam environment remain the same
CBT: What is different • An organization wanting to host an exam must fulfill more detailed criteria (basically, a set number of seats with reliable computers and Internet access with technical backup on hand for start of exam) • The proctor’s environment changes and proctor must enter a password before each candidate can start their exam
CBT: What is different • The candidate experience is much improved: – There is one question per page. It is much easier to read the question and answer options – Easy to leave questions unanswered and go back to them later or to answer them and ‘flag’ them for later review – You get the results within 5 business days
Three Exam Options – An exam hosted by an AIRS member (either at their own facility or at a community-based one) – An exam proctored by a college or university – An exam conducted by an online proctor via a webcam at home or at work
Hosted by AIRS member • Host must complete an application form (found via www.airs.org/certification) • Requirements include: – – – –
Facility with computers/laptops and Internet In a quiet setting Securing a proctor Having some technical expertise on hand on the day of the exam just in case
• No additional cost unless required by the host site to cover specific costs
Hosted by College/University • Exam can be taken at certain colleges in the United States (along with a few in Canada) that are members of the National College Testing Association (NCTA at www.nctatesting.org • Find an institution near you and go through the required steps • Additional cost usually of around $25 to $30 per person that is paid directly to the college
Hosted by Webcam-based Proctor
• AIRS working with company called ProctorU • Allows exam to be taken at home or at work (but in quiet/private setting) as long as you have webcam and secure Internet • Cost is $25 but provides flexibility and no travel costs
So What Does The Exam Look Like??
• https://ondemand.questionmark.com/deliver y/perception.php?customerid=399354
AIRS CBT: First Reports • The work involved in hosting an exam is a bit more involved for the AIRS member organization but (as with all other things), it will seem a lot simpler the second and third times around • The feedback from candidate has been very positive. It makes the exam process so much easier • However, we are working with technology and although it has not happened yet, something somewhere will go wrong – be patient and we will try our best to fix it
Questions • How come … ? • What about … ? • So what if … ?
And when you are ready, go to www.airs.org/certification If stuck or confused, email
[email protected]
