Detection of Advance Item Knowledge Using Response Times in Computer Adaptive Testing (CT-03-03)
by Rob R. Meijer and Leonardo S. Sotaridona, University of Twente, Enschede, The Netherlands

Executive Summary

The shift in test delivery from traditional paper-and-pencil (P&P) testing to computerized adaptive testing (CAT) was motivated by two main goals: increasing measurement efficiency and improving item security. Additional practical advantages of CAT are: the possibility of easier and more frequent test administration; immediate feedback of results; increased test taker interest; enhanced measurement precision; and reduced cost of test production, administration, and scoring.

Though item security was initially believed to be one of the greatest advantages of CAT, it soon became one of its major problems. To ensure item security, item banks needed to be continually updated. This necessity greatly increased the cost of implementing an operational CAT. A number of approaches to tackle the challenge of ensuring item security have been proposed.

In this paper, the use of information obtained from response times for detecting item security problems is investigated. We propose a new method based on an estimate of the “effective response time” for test takers on each item, which is based upon a method previously proposed by other researchers. Effective response time is defined as the time required for a test taker to answer an item correctly. An unusually short response time relative to the expected effective response time may be an indicator of item preknowledge. The test statistic in this new method was applied in an empirical study. Results showed that the false-positive error rate of the statistic could be controlled. Further analysis revealed that the detection rates produced by the statistic are sensitive to the magnitude of the reduction in response time as a result of item preknowledge.


Detection of Advance Item Knowledge Using Response Times in Computer Adaptive Testing (CT-03-03)

Research Report Index