Item Theft in a Continuous-Testing Environment: What is the Extent of the Danger? (CT-98-01)
by Deborah L. Schnipke, Law School Admission Council and David J. Scrams, Educational Testing Service

Executive Summary

Computerized test administration offers a number of advantages over traditional paper-and-pencil administrations, but some of these advantages may bring additional concerns as well. One often-mentioned advantage is the potential for continuous or on-demand testing. That is, test takers may have the freedom to schedule a test administration at a date and time convenient for the test taker rather than being required to take a group-administered test at one of a few prescheduled administrations spread throughout the year. The initial allure of continuous testing may be offset by the added risk to test security. The threat of one test taker relaying items to a friend may be a relatively minor issue, but the possibility of organized, large-scale item theft is far more serious. This issue was emphasized recently when Kaplan Educational Centers demonstrated that a small number of test takers could memorize items during a test administration and effectively memorize a significant portion of items from the computerized Graduate Record Examinations.

As the Law School Admission Council (LSAC) considers computerizing the Law School Admission Test (LSAT), the concerns that accompany the benefits of computerization must be addressed. The present study explores the dangers of item theft in terms of impact on test quality and fairness.

In this preliminary investigation, simulations were used to explore the impact of item theft on test taker scores. These simulations demonstrate how estimates of test taker ability are affected when test takers have access to some of the items in the item pool before they take the test. This access is facilitated by a simulated organized group of "thieves" who take the test, memorize the items received, and distribute these items to future test takers. Impact is explored for varying numbers of thieves as well as for thieves of two general ability levels: regular thieves who are similar in ability to average test takers, and professional thieves of substantially greater ability.

Results show that when regular thieves provided the stolen items, all but the highest ability test takers received inflated ability estimates. When professional thieves provided stolen items, some low-ability test takers were helped tremendously and received ability estimates at the top of the ability range. All test takers who had relatively high abilities received a substantial number of stolen items and also received ability estimates at the top of the ability range when professional thieves provided the stolen items.

Although the results may be cause for caution, concerns raised by the current work must be considered in the proper context. First, items and even complete test forms are occasionally stolen from current paper-and-pencil test administrations. Future test takers do not know when stolen pretest items will appear again, so the benefit of memorizing stolen items is reduced. The real problem of item theft is a concern introduced by continuous testing, not by computer administration. In a continuous testing environment, memorizing stolen items is of potentially substantial benefit if the item pool still contains the stolen items. A computer-administered test need not allow continuous testing, however, if all test takers can be accommodated with a set number of prescheduled test administrations. Second, the present study investigated only a very simple test design: a fully adaptive test with constraints only on item exposure. Additionally, only one item pool was used. Other designs may provide more opportunities to counteract the threat of item theft. One promising alternative is the multistage-testlet design currently being considered for a potential computerized LSAT administration. This and other possible solutions to the problem, such as item pool swapping, are discussed, and further research is planned to evaluate these possibilities.
 

Item Theft in a Continuous-Testing Environment: What is the Extent of the Danger? (CT-98-01)

Research Report Index