Do the National Tests of English in Norway test what we think they do? A think-aloud study of the eighth-grade tests from a validity perspective

Abstract

This article applies Messick’s (1995, 1996) presentation of the substantive aspect of validity (referred to as substantive validity) to the National Tests of English (NTE), taken by eighth-grade pupils in Norway. According to Messick, the substantive aspect is threatened if the test takers do not go through the processes intended/expected to correctly answer items. The article uses a think-aloud study to establish if the reading processes outlined in the tests’ accompanying documentation are evident when test-takers answer test items. Based on Khalifa and Weir’s (2009) hierarchy of reading processes, one of the models underpinning the NTE, ‘higher-level’ processes are associated with more difficult items. The study involved six eighth-grade pupils articulating their thoughts while answering a set of 30 NTE items, to establish if the items elicited the process levels intended by test-makers, thus supporting the argument for substantive validity.

The results showed that all the reading processes described in the tests’ accompanying documents were evident, at least to some extent, in the answers provided by the participants. However, of the correct answers provided, half were arrived at by the participants using processes at a lower level on the hierarchy than those intended by test-makers, indicating a threat to substantive validity. This threat may be tempered by a limited sample size and the fact that all participants at least used processes that were part of the tests’ framework. The article highlights the importance of think aloud studies to test development and, ultimately, in improving the strength of validity arguments, both for the NTE and for other large-scale language tests.