Automated essay assessment: an evaluation on paperrater’s reliability from practice / Nguyen Vi Thong

Nguyen, Vi Thong (2017) Automated essay assessment: an evaluation on paperrater’s reliability from practice / Nguyen Vi Thong. Creative Practices in Language Learning and Teaching (CPLT), 5 (1). pp. 1-18. ISSN 1823464-X

Abstract

From a perspective of a PaperRater user, the author attempts to investigate the reliability of the program. Twenty-four freshman students and one writing teacher at Dalat University - Vietnam were recruited to serve the study. The author also served as one scorer. The scores generated by PaperRater and the two human scorers were analyzed quantitatively and qualitatively. The statistical results indicate that there is an excellent correlation between the means of scores generated by three scorers. With the aid of SPSS and certain calculation, it is shown that PaterRater has an acceptable reliability which implies that the program can somehow assist in grading students’ papers. The semi-structured interview at the qualitative stage with the teacher scorer helped point out several challenges that writing teachers might encounter when assessing students’ prompts. From her perspective, it was admitted that with the assistance of PaperRater, the burden of assessing a bunch of prompts at a short time period would be much released. However, how the program can be employed by teachers should be carefully investigated. Therefore, this study provides writing teachers with pedagogical implications on how PaperRater should be used in writing classrooms. The study is expected to shed new light on the possibility of adopting an automated evaluation instrument as a scoring assistant in large writing classrooms.

Metadata

Item Type: Article
Creators:
CreatorsID Num. / Email
Nguyen, Vi Thongvithong1985@gmail.com
Subjects: L Education > LB Theory and practice of education > Computers in education. Information technology
L Education > LB Theory and practice of education > Systems of individual educators and writers
Divisions: Universiti Teknologi MARA, Kedah > Sg Petani Campus
Journal or Publication Title: Creative Practices in Language Learning and Teaching (CPLT)
Journal: UiTM Journal > Journal of Creative Practices in Language Learning and Teaching
ISSN: 1823464-X
Volume: 5
Number: 1
Page Range: pp. 1-18
Item ID: 30281
Uncontrolled Keywords: PaperRater. Writing assessment. Automated essay evaluation. Reliability. Human scorers
URI: http://ir.uitm.edu.my/id/eprint/30281

Download

[img] Text
AJ_NGUYEN VI THONG CPLT K 17.pdf

Download (638kB)

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year