Track №3 Assessing Student Learning Outcomes in Higher Education - Innovative Modeling and Measurement Approaches

Instructors:

Olga Zlatkin-Troitschanskaia, Full Professor, Chair of Business and Economics Education, JGU Mainz

Prof. Dr. Augustin Kelava, Head of the Method Center at the University of Tübingen

Abstract

The module "Assessing Student Learning Outcomes in Higher Education - Innovative Modeling and Measurement Approaches" focuses on the question how students’ domain-specific competencies and generic (interdisciplinary) skills (e.g., critical thinking) and their development over the course of higher education can be measured in a valid and reliable way. To allow for a valid and reliable measurement of students’ competencies and skills from the beginning to the end of their studies, theoretical-conceptual models and corresponding test instruments are necessary to describe and to measure the knowledge and skills of students in different phases of higher education (entering, undergraduate, graduate) with adequate precision and complexity.

The theoretical-conceptual background for this module is based on the Evidence-Centered Model for Instantiating Assessment Arguments by Riconscente and colleagues (2015). According to this model, a valid assessment allows for a psychometric connection between the construct that is to be measured (e.g., previous knowledge) and the developed tasks (e.g., subject-specific multiple-choice items). When assessing constructs such as students’ competencies, it is essential to determine which capabilities are to be assessed. Developing an appropriate task largely relies on the question in what kinds of situations a student would demonstrate the targeted competencies (Riconscente, Mislevy and Corrigan 2015; Mislevy 2018). Following this model, the module focuses first on the development of models to describe and operationalize students’ competencies and their development in higher education. The modeling process included the (e.g., multidimensional, content-related, cognitive) structure, taxonomy (e.g., difficulty levels), and development of (generic or domain-specific) competencies that are based on internationally established cognitive psychology, teaching-learning theory, domain-specificity and didactics. These competencies then served as a basis for the development of suitable measurement instruments for valid competency assessment. All of these different models follow the assumption that student competencies are complex, multi-faceted constructs (for a definition, see e.g. Weinert 2001). Depending on which competence facets and levels are targeted, for example, domain-specific knowledge or generic skills of beginning or advanced students, different measurement approaches are appropriate. Here, in the next step, the module focuses on a model developed by Blömeke, Gustafsson and Shavelson (2015), which describes students’ competencies across the continuum „dispositions“ – „skills“ – (observed) „performance“. Cognitive or affective-motivational dispositions (e.g., previous domain-specific knowledge or academic self-efficiency), which are considered latent traits of students, are often measured using multiple-choice tasks or self-report scales and are modeled in accordance with Item Response Theory. Situation-specific skills (e.g., teachers’ instructional skills) are typically measured using a situative approach (social-contextual roots) and, for instance, problem-based scenarios (e.g., assessments with video vignettes). Finally, performance is usually assessed using concrete, realistic, hands-on tasks sampled from criterion-related situations and analyzed using holistic approaches like generalizability theory models (e.g., Shavelson, Zlatkin-Troitschanskaia, Beck, Schmidt and Marino 2019).

In the next step, the module focuses on the development of the corresponding test instruments for the valid and reliable assessment of different facets and levels of students’ competencies using the above mentioned different assessment formats, which range from established selected-response questions with closed-answer formats to innovative computer-based performance assessments where the students have to act spontaneously.

Finally the module focuses on specific challenges and approaches with regard to assessing SLOs in higher education in cross-national studies.

Learning objectives

To familiarize participants with models to describe and operationalize students competencies and their development in higher education.

Learning outcomes

Know and use theoretical-conceptual models, corresponding test instruments to describe and to measure the knowledge and skills of students in different phases of higher education (entering, undergraduate, graduate) with adequate precision and complexity.

Be able to differ measurement approaches depending on the type of skills measured

Schedule of the track №3
Day 1.

Theoretical and conceptual background of assessing SLOs in higher education. Presentation of different assessment frameworks, focusing on the Evidence-Centered Model for Instantiating Assessment Arguments by Riconscente and colleagues (2015).

Day 2.

Developing competence models that differentiate and reliably describe the multidimensional competencies of students in different phases of higher education, focusing on content-related structure in a certain domain as wells as cognitive taxonomy.

Day 3.

Introduction to different measurement approaches, focusing on the model developed by Blömeke, Gustafsson and Shavelson (2015), which describes students’ competencies across the continuum „dispositions“ – „skills“– (observed) „performance“.

Day 4.

Development of suitable test instruments for the valid and reliable assessment of different facets and levels of students’ competencies using different assessment formats, focusing on innovative computer-based performance assessments.

Day 5.

Experiences from cross-national studies on student competencies in higher education and discussion of typical challenges, incl. bias due to translation and adaptation, as well as possible solution approaches that can make cross-cultural assessments more balanced, valid, and fair.