BORA - UiB

Bergen Open Research Archive

Guarding the Guardians Rating scale and rater training effects on reliability and validity of scores of an oral test of Norwegian as a second language

Bergen Open Research Archive

Show simple item record

dc.contributor.author Carlsen, Cecilie
dc.date.accessioned 2007-02-05T08:59:56Z
dc.date.available 2007-02-05T08:59:56Z
dc.date.issued 2004-01-16
dc.identifier.isbn 82-497-0187-9 (print version)
dc.identifier.uri http://hdl.handle.net/1956/2085
dc.description.abstract This thesis focuses on the scoring of a national test of Norwegian as a second language: Språkprøven i norsk for voksne innvandrere, developed by Norsk språktest at the University of Bergen. In order to ensure a fair assessment of the candidates’ oral production, the test constructors make use of trained raters basing their scores on an explicit rating scale (NORS). These two highly recommended procedures in performance testing have traditionally been viewed as means to heighten reliability of test scores. In line with recent developments in the field of language testing, I argue that the rater variable affects not only reliability, but the very construct validity of test scores. Rater training and development of rating scales are costly and time-consuming enterprises. To establish their effect on test scores is therefore interesting from a test theoretical, as well as from a practical and economical point of view. In the study, four groups of informants are compared: non-linguists (or naïve-native speakers), teachers of Norwegian as a second language without rater training, raters of Språkprøven, and finally a subgroup of the most experienced raters of Språkprøven. The informants score eight candidates’ video recorded performances on a six-point scale. The first four are scored impressionistically, and the next for by informants using the NORS. The quantitative data are used in an investigation of internal agreement (inter-rater reliability) between raters of the distinct groups. Informants are also asked to give written reports of their scores, which are used in an investigation of raters’ underlying criteria for assessing speech. The qualitative data are used firstly in an attempt to explain the results of the reliability study, and thereafter in an investigation of the match between raters’ criteria and the criteria of the NORS (construct validity). The results reveal differences between groups for the scores they give, as well as for the reasons for these scores. One important conclusion echoes the claim that “quantitative similarities in ratings may mask significant qualitative differences in the reasons for those ratings” (Connor-Linton 1995: 99). en
dc.format.extent 1035656 bytes
dc.format.mimetype application/pdf
dc.language.iso eng en
dc.publisher The University of Bergen en
dc.rights All rights reserved. en
dc.title Guarding the Guardians Rating scale and rater training effects on reliability and validity of scores of an oral test of Norwegian as a second language en
dc.type Doctoral thesis en
dc.subject.nsi VDP::Humaniora: 000::Språkvitenskapelige fag: 010::Nordiske språk: 018 no
dc.rightsHolder Cecilie Carlsen


Files in this item

 

This item appears in the following Collection(s)

Show simple item record

Search BORA


Browse

My Account