Investigating the Quality of Items in CAT Using Nonparametric IRT (CT-04-05)
by Rob R. Meijer, University of Twente, Enschede, The Netherlands

Executive Summary

The quality of the items in an item pool is an important determinant of the success of the computer adaptive testing (CAT) program. A mathematical model called item response theory (IRT) is used as the basis for many CAT programs, and statistics derived through IRT are among those that may be used to investigate the quality of the items in the item pool. Among the IRT models, a family of approaches referred to as nonparametric (NIRT) models are useful to investigate the quality of the items and response data because they are not based on strong functional assumptions and enable the use of informative data exploration techniques.

The aim of the present study is to illustrate the usefulness of NIRT for designing good item pools, a problem for which the solutions are still in their infancy. I show how the use of NIRT is very suitable for exploring the structure of CAT data. Particularly I explored the use of NIRT for analyzing the covariance structure between items as well as the (nonparametric) regression of item scores on total scores. It is shown that this use of NIRT leads to useful information, which (1) can be interpreted very easily by practitioners, (2) avoids forcing the data into a structure they sometimes do not have, and (3) is easily obtained through the use of very user-friendly software programs.
 

Investigating the Quality of Items in CAT Using Nonparametric IRT (CT-04-05)

Research Report Index