22 September 2020
Recently, a badly designed computer algorithm was implemented to award high school students grades due to the cancellation of national examinations. However, the algorithm failed spectacularly and awarded grades in an unfair manner, even penalizing students from poorer backgrounds and impoverished communities.
This monumental lapse threatened to derail the college prospects of untold numbers of students. Eventually, human-assigned grades were used.
The dangers of algorithms usually go unseen and undiscussed, because people lack the information and power they need to recognise and address them.
Many admissions and student services offices already use algorithms, typically to supplement humans. Those algorithms’ weaknesses and biases have the potential to proliferate broadly through the entire process of admitting students, distributing financial aid, grading, recommending classes or majors and surveilling students for other risks.
All algorithms should be seen as untrustworthy until proven otherwise. Until we as a society acknowledge this, and insist on the transparency required for the public to assess reliability and fairness, we’re not ready to use them.