Bergen Open Research Archive

Guarding the Guardians Rating scale and rater training effects on reliability and validity of scores of an oral test of Norwegian as a second language

Bergen Open Research Archive

Show simple item record Carlsen, Cecilie eng 2007-02-05T08:59:56Z 2007-02-05T08:59:56Z 2004-01-16 eng
dc.identifier.isbn 82-497-0187-9 (print version) eng
dc.description.abstract This thesis focuses on the scoring of a national test of Norwegian as a second language: Språkprøven i norsk for voksne innvandrere, developed by Norsk språktest at the University of Bergen. In order to ensure a fair assessment of the candidates’ oral production, the test constructors make use of trained raters basing their scores on an explicit rating scale (NORS). These two highly recommended procedures in performance testing have traditionally been viewed as means to heighten reliability of test scores. In line with recent developments in the field of language testing, I argue that the rater variable affects not only reliability, but the very construct validity of test scores. Rater training and development of rating scales are costly and time-consuming enterprises. To establish their effect on test scores is therefore interesting from a test theoretical, as well as from a practical and economical point of view. In the study, four groups of informants are compared: non-linguists (or naïve-native speakers), teachers of Norwegian as a second language without rater training, raters of Språkprøven, and finally a subgroup of the most experienced raters of Språkprøven. The informants score eight candidates’ video recorded performances on a six-point scale. The first four are scored impressionistically, and the next for by informants using the NORS. The quantitative data are used in an investigation of internal agreement (inter-rater reliability) between raters of the distinct groups. Informants are also asked to give written reports of their scores, which are used in an investigation of raters’ underlying criteria for assessing speech. The qualitative data are used firstly in an attempt to explain the results of the reliability study, and thereafter in an investigation of the match between raters’ criteria and the criteria of the NORS (construct validity). The results reveal differences between groups for the scores they give, as well as for the reasons for these scores. One important conclusion echoes the claim that “quantitative similarities in ratings may mask significant qualitative differences in the reasons for those ratings” (Connor-Linton 1995: 99). en
dc.format.extent 1035656 bytes eng
dc.format.mimetype application/pdf eng
dc.language.iso eng eng
dc.publisher The University of Bergen eng
dc.rights All rights reserved eng
dc.title Guarding the Guardians Rating scale and rater training effects on reliability and validity of scores of an oral test of Norwegian as a second language eng
dc.type Doctoral thesis eng
dc.subject.nsi VDP::Humaniora: 000::Språkvitenskapelige fag: 010::Nordiske språk: 018 nob
dc.rights.holder All rights reserved
dc.rights.holder Cecilie Carlsen eng

Files in this item


This item appears in the following Collection(s)

Show simple item record

Search BORA


My Account