Dealing with multiple objectives in test assembly
Bernard P Veldkamp
Research Center for Examination and Certification
University of Twente - The Netherlands
University of Bologna - Italy
Requests for information can be send to: Bernard Veldkamp, University of Twente, PO box 217, 7500 AE Enschede, The Netherlands. E-mail: email@example.com
It is quite common that tests or exams are being used for more then one purpose. For example, they are used to measure the ability of the students in a reliable manner. At the same time, they are used for pass/fail decisions or to predict future behavior of the candidate. The question remains how to assemble a test that can be used for all these different purposes, that is, how to assemble a multi-objective test. Besides, multiple objectives can result from different purposes, but also from the way test specifications have been implemented. For the WDM-model, for multidimensional IRT, for Cognitive Diagnostic CAT, but also for infeasibility analysis, multiple objective test assembly problems have to be solved.
In this paper, a 2-stage method is presented for dealing with multiple objectives in test assembly. In the normalization stage, all objectives are brought on a common scale. In the valorization stage, the different objectives are being compared and related to each other. The method is applied to a Guidance Test developed at the University of Bologna, and a comparison is made with more traditional single objective test assembly methods. The results clearly demonstrate the importance and relevance of multi-objective test assembly.
Keywords: automated test assembly, multiple criteria, multiple objectives, simulated annealing, guidance test.