Track №2 "Analyzing Rating Data Using Many-Facet Rasch Measurement and Multilevel Rater Modelling"

Affiliation: Wilbur Wright College

Course Description

The “Analyzing Rating Data Using Many-Facet Rasch Measurement and Multilevel Rater Modelling” course by Dr. Lidia Dobria introduces participants to two approaches for analyzing performance assessment data: the many-facet Rasch measurement approach and the multilevel rater model approach. In the context of performance assessments, raters who evaluate students’ performances or products will unavoidably introduce errors (i.e., rater effects) into the process. The statistical approaches discussed in this course will help model and investigate the performance of various facets (e.g., students, items, raters) of the assessment system. The information gleaned in the process can help determine to what extent the system is under statistical control and guide subsequent efforts for improvement.

The course consists of two parts. The first will focus on the many-facet Rasch model (MFRM) for analyzing rating data, and the second will feature the multilevel rater model (MRM) – a recasting of the MFRM within the multilevel framework. The topics covered in the lectures will be exemplified using real data sets, and each session will be followed by hands-on demonstrations.

The topics covered during the MFRM portion of the training are as follows:

The role and influence of raters in open-ended assessments
Rater effects that can influence ratings
Conceptual and mathematical foundations of many-facet Rasch measurement
Group and individual-level research questions that can be answered using an MFRM analysis
Creating a specification file for an MFRM analysis using the Minifac/Facets software package
Interpreting and presenting output from MFRM analyses
Creating judging plans
MFRM anchoring procedures for equating
Strategies for resolving disconnected subsets issues
Bias interaction analyses
Hybrid MFRM models and their uses

The topics covered during the MRM portion of the training include the following:

Conceptual and mathematical foundations of the multilevel model framework
Reformulating the MFRM model as a multilevel rater model
Preparing data files required for a multilevel analysis
Running MRM analyses using the HLM software package
Interpreting and presenting output from MRM analyses
Extending the MRM by including additional random effects and/or covariates of interest
Carrying out differential facet functioning analyses in the multilevel context

Prerequisites

The course requires that participants have prior experience with multiple regression and analysis of variance. A basic understanding of the fundamentals of Rasch measurement would be helpful but is not required, as would prior experience with Rasch analysis. Similarly, a basic understanding of the hierarchical linear modeling framework would be helpful but is not required, as would prior experience using hierarchical linear modeling software.

Learning Objectives

Participants will work toward acquiring the following:

An understanding of the role of raters and how they influence ratings in open-ended assessments
An understanding of different types of rater effects
Knowledge of conceptual and mathematical foundations of the MFRM and MRM.
Knowledge of the advantages and disadvantages of using the MFRM and MRM to analyze rating data
Knowledge of group and individual-level research questions that can be addressed using the MFRM and MRM
Skill in creating judging plans
Skill in creating specification files for MFRM and MRM analyses
Skill in running analyses of rating data using different software packages for rater data (Facets/Minifac, HLM)
Skill in interpreting MFRM and MRM analysis output

Learning Outcomes

Understand the theoretical underpinnings of different frameworks for analyzing rating data.
Gain practical experience running different MFRM and MRM models and interpreting the resulting output.
Develop baseline competencies in running different software packages for the analysis of rating data.

References

Linacre, J. M. (2022) Facets computer program for many-facet Rasch measurement, version 3.84.0. Beaverton, Oregon: Winsteps.com.

Raudenbush, S.W., & Congdon, R.T. (2021). HLM 8: Hierarchical linear and nonlinear modeling. Chapel Hill, NC: Scientific Software International, Inc.