Automated essay assessment: an evaluation on paperrater’s reliability from practice / Nguyen Vi Thong

Nguyen, Vi Thong (2017) Automated essay assessment: an evaluation on paperrater’s reliability from practice / Nguyen Vi Thong. Journal of Creative Practices in Language Learning and Teaching (CPLT), 5 (1). pp. 1-18. ISSN 1823-464X

Abstract

From a perspective of a PaperRater user, the author attempts to investigate the reliability of the
program. Twenty-four freshman students and one writing teacher at Dalat University - Vietnam
were recruited to serve the study. The author also served as one scorer. The scores generated by
PaperRater and the two human scorers were analyzed quantitatively and qualitatively. The
statistical results indicate that there is an excellent correlation between the means of scores
generated by three scorers. With the aid of SPSS and certain calculation, it is shown that
PaterRater has an acceptable reliability which implies that the program can somehow assist in
grading students’ papers. The semi-structured interview at the qualitative stage with the teacher
scorer helped point out several challenges that writing teachers might encounter when assessing
students’ prompts. From her perspective, it was admitted that with the assistance of PaperRater,
the burden of assessing a bunch of prompts at a short time period would be much released.
However, how the program can be employed by teachers should be carefully investigated.
Therefore, this study provides writing teachers with pedagogical implications on how
PaperRater should be used in writing classrooms. The study is expected to shed new light on the
possibility of adopting an automated evaluation instrument as a scoring assistant in large
writing classrooms.

Metadata

Item Type: Article
Creators:
Creators
Email / ID Num.
Nguyen, Vi Thong
vithong1985@gmail.com
Subjects: L Education > LB Theory and practice of education > Computers in education. Information technology
L Education > LB Theory and practice of education > Systems of individual educators and writers
Divisions: Universiti Teknologi MARA, Kedah > Sg Petani Campus
Journal or Publication Title: Journal of Creative Practices in Language Learning and Teaching (CPLT)
UiTM Journal Collections: UiTM Journal > Journal of Creative Practices in Language Learning and Teaching (CPLT)
ISSN: 1823-464X
Volume: 5
Number: 1
Page Range: pp. 1-18
Keywords: PaperRater. Writing assessment. Automated essay evaluation. Reliability. Human scorers
Date: 2017
URI: https://ir.uitm.edu.my/id/eprint/30281
Edit Item
Edit Item

Download

[thumbnail of AJ_NGUYEN VI THONG CPLT K 17.pdf] Text
AJ_NGUYEN VI THONG CPLT K 17.pdf

Download (638kB)

ID Number

30281

Indexing

Statistic

Statistic details