Dr. Kevin Raczynski recently published an article in the Journal of Educational Measurement entitled “Comparing the effectiveness of self-paced and collaborative frame-of-reference training on rater accuracy in large scale writing assessment”.
This article described research performed by Dr. Raczynski that looked at different methods of rater training; specifically, this study included a comparison of self-paced rater training and collaborative rater training.
Dr. Raczynski randomly assigned 60 experienced raters into two groups, training the first group by a self-paced method, and the second group by a collaborative method. The goal of the research was to determine which method yielded better results in terms of rater accuracy.
Dr. Raczynski theorized that the collaborative method would produce better accuracy among raters, in part because of the different viewpoints each rater brought to the discussion. However, it was found that there was not a significant difference in rater accuracy between the two methods of scoring.
Dr. Raczynski’s article and conclusions can be found here: