The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

Bethany Fishbein · Michael O. Martín · Ina V. S. Mullis · Pierre Foy

52 citations0/0 files storedLarge-scale Assessments in Education

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Psychometric Methodologies and Testing Educational Assessment and Pedagogy Educational Technology and Assessment

Abstract

TIMSS 2019 is the first assessment in the TIMSS transition to a computer-based assessment system, called eTIMSS. The TIMSS 2019 Item Equivalence Study was conducted in advance of the field test in 2017 to examine the potential for mode effects on the psychometric behavior of the TIMSS mathematics and science trend items induced by the change to computer-based administration. The study employed a counterbalanced, within-subjects design to investigate the potential for eTIMSS mode effects. Sample sizes for analysis included 16,894 fourth grade students from 24 countries and 9,164 eighth grade students from 11 countries. Following a review of the differences of the trend items in paper and digital formats, item statistics were examined item by item and aggregated by subject for paperTIMSS and eTIMSS. Then, the TIMSS scaling methods were applied to produce achievement scale scores for each mode. These were used to estimate the expected magnitude of the mode effects on student achievement. The results of the study provide support that the mathematics and science constructs assessed by the trend items were mostly unaffected in the transition to eTIMSS at both grades. However, there was an overall mode effect, where items were more difficult for students in digital formats compared to paper. The effect was larger in mathematics than science. Because the trend items cannot be expected to be sufficiently equivalent across paperTIMSS and eTIMSS, it was concluded that modifications must be made to the usual item calibration model for TIMSS 2019 to measure trends. Each eTIMSS 2019 trend country will administer paper trend booklets to a nationally representative sample of students, in addition to the usual student sample, to provide a bridge between paperTIMSS and eTIMSS results.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 52 citations exist upstream.

References

1983Developing a Common Metric in Item Response Theory 1996Science Achievement in the Middle School Years : Iea's Third International Mathematics and Science Study (Timss)2006The digital divide: the special case of gender 1986Guidelines for computer-based tests and interpretations 2014A comparison of reading comprehension across paper, computer screens, and tablets: Does tablet familiarity matter?2014Understanding tablet computer usage among primary school students in underdeveloped areas: Students’ technology experience, learning styles and attitudes 1988THE EQUIVALENCE OF SCORES FROM AUTOMATED AND CONVENTIONAL EDUCATIONAL AND PSYCHOLOGICAL TESTS 2004Developing Computerized Versions of Paper-and-Pencil Tests: Mode Effects for Passage-Based Tests

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.