Above: Steve Marostica and Savanna Bond – Scoring Team Leads at GCA
Spring is not just a season known for blooming flowers and the nuisance of pollen, it is also one of the busiest seasons in scoring for GCA. Inside our building, a former practice facility for the Olympics, is an oversized room that houses a sea of computers, ready to be used by college-educated people. These people – often referred to as “raters,” “evaluators,” or “scorers” – will settle into designated areas in mid-April as they prepare to complete important, confidential work. While the work is fulfilling, it is not for the faint at heart. It requires an acute ability to stay focused and remain unbiased. The work? Scoring student responses for high-stakes summative assessments.
GCA is no stranger to assessment scoring. We’ve been scoring writing since the late 80’s and we have no plans of slowing down! But we can’t take all the credit. The key to our success lies in the people that we hire and the hands-on staff who steer the projects from beginning to end.
Finding quality candidates starts with a well-planned and executed recruiting process led by our Scoring and Reporting unit. While experience is not required for potential raters, one thing is: a 4-year college degree or better. Candidates go through an interview process that includes a test run of the scoring process as well as responses to scenario-based questions. This is critical because we can pinpoint scoring habits that will likely transfer to live scoring. If the results of the mock scoring session are favorable, the candidate moves forward in the hiring process which includes an extensive background check to verify credentials.
Even after candidates are fully hired, we continue our efforts of “vetting” candidates. We do this by administering a qualifying test. This occurs after an extensive training session in which Scoring Directors walk raters through an item with pre-scored anchor sets and student responses that include explanatory notes that become models of appropriate answers for each scoring level. A scoring rubric is also reviewed in detail to determine where a student response falls according to a defined set of criteria. This will help ensure raters assign accurate and fair scores to each response.
Once a rater has qualified, live scoring officially begins, but GCA’s high standards don’t stop there! Our Scoring Directors and Team Leads, referred to as SDs and TLs, read behind raters’ scores to ensure they fall within acceptable ranges. If a rater is scoring too high or too low (based on the rubric) on any given day, SDs and TLs provide guidance to get the rater back on track. Measures are also in place to check a rater’s accuracy by including sporadic responses called validity. These responses are pre-scored and help the team quickly determine rater tendencies.
Productivity is just as critical as accuracy due to the large number of responses that need to be scored daily. SDs run reports to ensure each rater is producing an acceptable amount of work compared to other raters. Rater agreement is also closely monitored from day-to-day. And our raters are never alone! Help is available and training materials are always at their fingertips.
GCA is proud to provide individuals with a solid short-term opportunity each spring. Raters who perform well on our projects are asked to return, and they do! We value each Scoring Director, Team Lead, and rater who joins the GCA team. Our mission is to continue this path of providing top-notch hand-scoring services to schools, districts, states, and even countries!