THE PREDICTION OF TOEFL READING COMPREHENSION ITEM DIFFICULTY FOR EXPOSITORY PROSE PASSAGES FOR THREE ITEM TYPES: MAIN IDEA, INFERENCE, AND SUPPORTING IDEA ITEMS

Abstract

ABSTRACT The purpose of the current study is to predict the difficulty (equated delta) of a large sample (n=21.3) of TOEFL reading comprehension items. (Only main idea, inference, and supporting statement items were sampled.) A related purpose was to examine whether text and text‐related variables play a significant role in predicting item difficulty; we argued that evidence favoring construct validity would require significant contributions from these particular predictor variables. In addition, details of item predictability were explored by evaluating two hypotheses: (1) that multiple‐choice reading comprehension tests are sensitive to many sentential and discourse variables found to influence comprehension processes in the experimental literature, and (2) that many of the variables identified in the first hypothesis contribute significant independent variance in predicting item difficulty. The great majority of sentential and discourse variables identified in our review of the experimental literature were found to be significantly related to item difficulty within TOEFL's multiple‐choice format. Furthermore, contrary to predictions which we attributed to critics of multiple‐choice tests, the pattern of correlational results showed that there is a significant relationship between item difficulty and the text and text‐related variables. We took this as evidence supporting our claim that multiple–choice reading items yield construct valid measures of comprehension. That is, since critics have pointed out that reading items can often be correctly answered without reading of the text passage, this seems to imply that item variables (not text nor text‐related variables) should fbe prominent predictors of reading item difficulty. Since the contrary relationship was found, we concluded that this provides evidence favoring construct validity. We found, further, in several stepwise linear regression analyses, that many of these text and taxt‐related variables provide independent contributions in predicting reading item difficulty. This was interpreted as providing additional support for construct validity. More specifically, apart from the correlational results, the following stepwise linear regressions results were obtained. For the full sample of 213 items, and where equated delta (an index of item difficulty) is the dependent variable, we found 33 percent (p < .0001) of the variance of item difficulty could be accounted for by sight variables. All sight variables reflected significant and independent contributions due solely to text and text/item overlap variables. This result provided evidence favoring constract validity of the TOEFL reading comprehension items. We also conducted a separate analysis of a subset (n=98) of the full set of 213 items to examine the possible statistical effect of nesting in the original sample, (Nesting occurs when several items relating to the same passage are analysed together; a non‐nested subset is formed when only one item per passage is used.) Eleven variables accounted for 58 percent. (p < .0001) of the variance of this non‐nested sample. Ten of these 11 variables reflected significant and independent contribution of text and text/item overlap variables. Eencs this subanalysis provided further support for construct validity. While both analyses provide evidence favoring construct validity of the TOEFL reading items, the differences in amount of variance accounted for suggests that nesting affects should be a concern in future studies predicting item difficulty.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 28 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.