Methods for making classification decisions, Maaike van Groen

Most adaptive tests are constructed in order to estimate the examinees’ ability as efficient and accurate as possible. Computerized classification testing (CCT) has a different goal: classify the examinee as efficient and accurate as possible into mutual exclusive groups. A classification decision is made in which the examinee is assigned into one of two or more mutually exclusive categories along the ability scale (Lin & Spray, 2000) using cutting points to separate the categories (Eggen, 1999).

A computerized classification test is of variable length and examinees ‘’are classified as masters or non-masters as soon as there is enough evidence to make a decision’’ (Finkelman, 2008). The classification procedure must choose between three options: to stop testing and classify an examinee as a master, to stop testing and classify an examinee as a non-master, or to continue testing and select a new item. Several procedures are available for making the decisions but also for the way in which items are selected.

The sequential mastery test procedure has to determine whether it is possible to classify an examinee. The examinee is classified using some statistical rule which bases its decision on certain parameters (Thompson, 2009). Thompson (2009) has divided the termination methods into three types which all cover a range of comparable statistical procedures for termination decisions. The sequential probability ratio testing (SPRT) procedure uses hypothesis testing for classification of the respondents. The ability confidence intervals procedure, which was developed by Kingsbury and Weiss (1979; ACI), constructs intervals around the current ability estimate (Thompson, 2009). If the cut point is outside the interval a classification decision can be made. DeGroot (1970) has formulated a Bayesian decision theory for making statistical decisions. Rudner (2002) and Vos (2000) applied decision theory for making classifications of respondents. The objective of decision theory in the context of computerized classification testing is to make a best guess of the level of an examinee based on responses, a priori item information, and a priori population classification proportions (Rudner, 2002).

An item selection method selects an item using the current test performance of the examinee (Chang & Ying, 1996). In CAT, item selection methods are often based on maximization of a concept called information at the current estimate of the respondent’s ability. In CCT, item selection by selecting the item with the most information at the current estimate of the examinee’s ability is used frequently, but other approaches are also popular including maximizing information at the cut point. The value of Fisher information at the true ability level of an examinee reflects the efficiency of the item for estimation of that ability or said differently the information is inversely related to the error of the estimated ability (Chang & Ying, 1996). Other methods use a Bayesian approach (Owen,1975). Inclusion of exposure mechanisms and content constraints in the item selection procedures is also possible.

The major advantage of adaptive tests is that more efficient ability estimates are provided using fewer items than required by conventional tests (Weiss, 1982).  Shorter tests reduce the testing time, the costs for the testing company, lessens the exposure rates of some items, reducing problems with test security and finally it reduces the frequency in which the item pool has to be changed (Finkelman, 2008). A shorter test is assembled for respondents who have clearly attained a certain level and longer tests for students for whom the decision is not as clear-cut. A second major advantage is that items are administered at the level of the respondents which increases examinees’ motivation and reduces the number of too easy or too difficult items that have to be answered by the examinees.

Although several item selection methods and methods for making the classification decision have been developed the focus of this research project will be on the development of new and investigation of current item selection methods and methods for making classification decisions. The following topics have been selected for the study: