Does your surname start with one of the last letters in the English alphabet? If it wasn’t already bad enough having to wait your turn in roll calls, it turns out you could also be more likely to receive lower marks and negative feedback on assignments.
Those are the findings of a team from the University of Michigan, who analyzed more than 30 million student records of assessments submitted and graded via a learning management system (LMS) called Canvas.
The researchers found this sequential bias occurs university-wide, regardless of subject, but is most pronounced in social science and humanities, impacting subjects like engineering, science, and medicine less. This might be due to the fact that assignments in the social sciences are more open to interpretation and therefore more difficult to grade, in contrast to engineering, science, or medicine, which tend to have more concrete and specific answers.
While this phenomenon is linked to alphabetization because of Canvas’s default settings, the research reveals a clear underlying pattern: grading quality declines as graders proceed through assignments.
Students who are marked earlier in the piece because their surnames begin with A, B, C, D or E, were given grades 0.3 points higher (out of 100 possible points) than what they received when grading order was randomized.
And students with surname initials U to Z were given grades 0.3 points lower when marking was done in alphabetical order, than when it was randomized.
But even outside of alphabetization, order mattered. The first ten assignments marked were usually given around 3.5 per 100 points more than those graded 50th to 60th.
And among the small group of graders that marked in reverse alphabetical order (Z-A), the surname initial bias was almost exactly reversed, meaning those with A-E surnames were worse off than their U-Z peers.
Before you blame your teachers, the researchers point out that a key factor in all this is grader fatigue. It’s not surprising a grader might get exhausted and a bit snappy towards the end of the marking period: in many places in the world, academics are over-worked, underpaid and lack job security.
“We kind of suspect that fatigue is one of the major factors that is driving this effect, because when you’re working on something for a long period of time, you get tired and then you start to lose your attention and your cognitive abilities are dropping,” says computational social scientist Jiaxin Pei.
Wang and colleagues suggest education institutions may need to hire additional graders to lighten staff workloads and cross-validate marks, to reduce the possible impacts of grader fatigue.
It was unclear whether lower grades necessarily meant lower grading quality, so the team analysed the graders’ comments, and students’ post-assessment queries and requests for regrades, to rule out the possibility that graders were simply becoming better at detecting errors as they progressed through marking.
They found graders’ comments tended to be more negative and less polite if the assignment was graded later, and that the results of these assignments were more likely to be disputed.
This “directly indicates that the grading quality is indeed lower for later-graded assignments and students with later surnames,” the authors write.
Canvas is the most widely-used LMS worldwide, and alphabetical order is its default mode for ordering student assignments for grading. The researchers say the simplest solution to addressing sequential bias would be making randomized ordering the default setting.
“The system design of the LMS converts individual-level grading bias into widespread disparity against students with surnames late in the alphabet,” the researchers write.
This working paper is yet to be published, but is under review by Management Science.
