1st Edition

Computerized Multistage Testing
Theory and Applications




ISBN 9781466505773
Published May 7, 2014 by Chapman and Hall/CRC
546 Pages - 90 B/W Illustrations

USD $105.00

Prices & shipping based on shipping country


Preview

Book Description

Devising tests that evaluate a nation’s educational standing and implement efficacious educational reforms requires a careful balance among the contributions of technology, psychometrics, test design, and the learning sciences. Unlike other forms of adaptive testing, multistage testing (MST) is highly suitable for testing educational achievement because it can be adapted to educational surveys and student testing. Computerized Multistage Testing: Theory and Applications covers the methodologies, underlying technology, and implementation aspects of this type of test design.

The book discusses current scientific perspectives and practical considerations for each step involved in setting up an MST program. It covers the history of MST, test design and implementation for various purposes, item pool development and maintenance, IRT-based and classical test theory-based methodologies for test assembly, routing and scoring, equating, test security, and existing software. It also explores current research, existing operational programs, and innovative future assessments using MST.

Intended for psychologists, social scientists, and educational measurement scientists, this volume provides the first unified source of information on the design, psychometrics, implementation, and operational use of MST. It shows how to apply theoretical statistical tools to testing in novel and useful ways. It also explains how to explicitly tie the assumptions made by each model to observable (or at least inferable) data conditions.

Winner of the 2016 AERA Award for Significant Contribution to Educational Measurement and Research Methodology
The 2016 American Education Research Association (AERA) Div. D award committee for Significant Contributions to Educational Measurement and Research Methodology has recognized unanimously this collaborative work advancing the theory and applications of computerized MST. This annual award recognizes published research judged to represent a significant conceptual advancement in the theory and practice of educational measurement and/or educational research methodology. The 2016 award was made under the heading: Measurement, Psychometrics, and Assessment. This collective work, published in 2014 as an edited volume titled Computerized Multistage Testing: Theory and Applications, was cited by the committee both for the originality of the conceptual foundations presented in support of multistage testing and for arguing persuasively for its potential impact on the practice of educational measurement.

Table of Contents

Test Design, Item Pool, and Maintenance
Overview of Computerized Multistage Tests Duanli Yan, Charles Lewis, and Alina A. von Davier
Linear Tests and Computerized Adaptive Tests (CATs)
Multistage Tests (MSTs)
MST Designs for Different Purposes
Implementation Schemes
Designing MST
Content Balance and Assembly
Exposure Control
Routing
Scoring, Linking, and Equating
Reliability, Validity, Fairness, and Test Security
Current and Future Applications
Logistic Challenges
Summary

Multistage Test Designs: Moving Research Results into Practice April L. Zenisky and Ronald K. Hambleton
The MST Design Structure
The State of Research: MST Development and Design Considerations
Conclusions and Next Steps

Item Pool Design and Maintenance for Multistage Testing Bernard P. Veldkamp
Designing an Item Pool Blueprint
Applications in Item Writing
Maintenance
Discussion

Mixed-Format Tests: Issues and Methods Jiseon Kim and Barbara G. Dodd
Literature Review on Design Components in Mixed-Format MST
Comparing Other Testing Approaches
Issues and Future Research Suggestions for Mixed-Format MST
Conclusion

Design and Implementation of Large-Scale Multistage Testing Systems Richard Luecht
MST Design and Implementation Considerations
Conclusions: A Research Agenda

Test Assembly
Overview of Test Assembly Methods in Multistage Testing
Yi Zheng, Chun Wang, Michael J. Culbertson, and Hua-Hua Chang
MST Framework
MST Assembly Design
Automated Assembly for MST
Setting Difficulty Anchors and Information Targets for Modules
"On-the-Fly" MST (OMST) Assembly Paradigm
MST, CAT, and Other Designs—Which Way to Go?

Using a Universal Shadow-Test Assembler with Multistage Testing Wim J. van der Linden and Qi Diao
Solving Shadow-Test Assembly Problems
Basic Design Parameters
Different Testing Formats
Relative Efficiency of Formats
Empirical Study
Concluding Comments

Multistage Testing by Shaping Modules on the Fly Kyung (Chris) T. Han and Fanmin Guo
MST by Shaping
MST-S versus MST-R versus CAT
Discussion and Conclusion

Optimizing the Test Assembly and Routing for Multistage Testing Angela Verschoor and Theo Eggen
Optimizing MST Assembly: A Nonexhaustive Search
Limited Item Pools, Two- and Three-Parameter Models
Discussion

Routing, Scoring, and Equating
IRT-Based Multistage Testing
Alexander Weissman
Introduction
Motivation for Tailored Testing
Routing Rules
Scoring and Classification Methodologies
Final Comments

A Tree-Based Approach for Multistage Testing Duanli Yan, Charles Lewis, and Alina A. von Davier
Regression Trees
Tree-Based Computerized Adaptive Tests
Tree-Based Multistage Testing
Algorithm
An Application
Discussion
Limitations and Future Research

Multistage Testing for Categorical Decisions Robert Smith and Charles Lewis
Computer-Mastery Methods
Information Targeted at Cut versus at Ability
Influence of Multiple Cut Scores
Factors That Can Reduce Optimal Solutions
Example Based on Smith and Lewis (1995)

Adaptive Mastery Multistage Testing Using a Multidimensional IRT Model C.A.W. Glas
Introduction
Definition of the Decision Problem
Computation of Expected Loss and Risk Using Backward Induction
Selection of Items and Testlets
Simulation Studies
Conclusions and Further Research

Multistage Testing Using Diagnostic Models Matthias von Davier and Ying (Alison) Cheng
The DINA Model and the General Diagnostic Model
Experience with CD-CATs
CD-MSTs
Discussion

Considerations on Parameter Estimation, Scoring, and Linking in Multistage Testing Shelby J. Haberman and Alina A. von Davier
Notation
The Item Response Model
The Test Score
Approaches to Parameter Estimation
Conclusions

Test Reliability, Validity, Fairness, and Security
Reliability of Multistage Tests Using Item Response Theory
Peter W. van Rijn
Test Reliability
Application: IRT Reliability for MST in NAEP
Conclusion

Multistage Test Reliability Estimated via Classical Test Theory Samuel A. Livingston and Sooyeon Kim
The Estimation Procedure
Testing the Accuracy of the Estimation Procedure
How Accurate Were the Estimates?

Evaluating Validity, Fairness, and Differential Item Functioning in Multistage Testing Rebecca Zwick and Brent Bridgeman
Content Balancing
Opportunities for Item Review and Answer Changing
Skipping Strategies
MST Routing Algorithms
The Digital Divide
Comparability of Computer Platforms
Accommodations for Students with Disabilities and English Language Learners
Differential Item Functioning Analysis in MSTs
Application of the Empirical Bayes DIF Approach to Simulated MST Data
Summary

Test Security and Quality Control for Multistage Tests Yi-Hsuan Lee, Charles Lewis, and Alina A. von Davier
An Overview of a Three-Component Procedure
Tools to Evaluate Test Security and Quality Control
A Simulation Study Using CUSUM Statistics to Monitor Item Performance
Discussion

Applications in Large-Scale Assessments
Multistage Test Design and Scoring with Small Samples
Duanli Yan, Charles Lewis, and Alina A. von Davier
Small Data Sample
Item Pool
Various MST Module Designs
Routing and Scoring
Comparisons of the Six MST Designs
Discussion
Limitations and Future Research

The Multistage Test Implementation of the GRE Revised General Test Frédéric Robin, Manfred Steffen, and Longjuan Liang
From CAT to MST
MST Design
Implementation
Monitoring
Summary

The Multistage Testing Approach to the AICPA Uniform Certified Public Accounting Examinations Krista J. Breithaupt, Oliver Y. Zhang, and Donovan R. Hare
Research on Multistage Testing
Item Bank Development for MST
Content Security Monitoring for MST
Inventory Exposure Planning for MST
Discussion

Transitioning a K–12 Assessment from Linear to Multistage Tests Carolyn Wentzel, Christine M. Mills, and Kevin C. Meara
Administering CTP Items Online
Creating a New MST Scale Using IRT
Multistage-Adaptive Test Development
Score Reporting
Summary

A Multistage Testing Approach to Group-Score Assessments Andreas Oranje, John Mazzeo, Xueli Xu, and Edward Kulick
Targeted Testing
Goals of the Study
Methods
Results
Discussion

Controlling Multistage Testing Exposure Rates in International Large-Scale Assessments Haiwen Chen, Kentaro Yamamoto, and Matthias von Davier
Item Exposure Rate Control for Multistage Adaptive Assessments
Method: How to Compute and Adjust the Item Exposure Rates
Data
Results
Conclusion

Software Tools for Multistage Testing Simulations Kyung (Chris) T. Han and Michal Kosinski
MSTGen
R
Conclusions

Closing Remarks
Past and Future of Multistage Testing in Educational Reform
Isaac I. Bejar
Future of MST
A Model-Based Three-Stage Design
Item Generation and Automated Scoring and Broadly Accessible Test Content
Summary and Conclusions

Bibliography

...
View More

Reviews

"… this is a terrific book and the editors should be congratulated."
Psychometrika, Vol. 80, No. 1, March 2015