US colleges and universities can shape well-rounded and diverse student bodies by using artificial intelligence tools to assess admissions essays, according to a University of Pennsylvania-led research team that conducted a test run on more than 300,000 applications.
The researchers submitted 150-word application essays from the students into a program that hunted for signs of seven different personal qualities, and said the process produced holistic assessments of the students with “minimal bias” and with overall accuracy in predicting eventual graduation success.
“The finding,” said a lead author, Angela Duckworth, a professor of psychology at Penn, “is that we can reliably detect personal qualities” using AI tools.
AI has been changing or threatening to change established practices across much of society, including higher education. In terms of college admissions, the questions and concerns have largely focused on how institutions could or should address the challenge of students using AI technologies to write their admissions essays. Some campus admissions offices also have begun working with AI-powered tools, mostly to cull and compare basic data such as high-school grade scores, although some have taken steps in the direction of using AI to synthesise large volumes of other applicant submissions materials including essays.
The Penn-led study involved nearly 310,000 students who applied to US institutions nationwide in 2008 and 2009. The research team had colleagues from the University of Colorado at Boulder, the University of Denver and Virginia Tech, and their findings were published in the peer-reviewed journal Science Advances.
Among the advantages of using AI tools in admissions, Professor Duckworth said, is the possibility of allowing more transparency in the process – if institutions are willing to make public the details of their systems.
The analysis also comes as some academic leaders and policymakers, including US education secretary Miguel Cardona, have been suggesting that selective institutions make greater use of application essays to identify and boost racial minorities in the aftermath of the US Supreme Court ruling against race-based preferences in admissions.
The Penn assessment – which used a language model known as a Robustly Optimized BERT Pretraining Approach, or RoBERTa – did not investigate whether the AI-driven essay reviews could be used for the specific purpose of aiding racial diversity. Instead, the study watched the AI tool compare applicants on seven personal qualities – prosocial purpose, leadership, teamwork, learning, perseverance, intrinsic motivation and goal pursuit – and found what the study team regarded as no bias in how those qualities were judged.
Despite the Penn findings, some outside experts have questioned the value of essays in student admissions and don’t see AI technology as fixing the associated problems. A study published in 2021 in Science Advances and led by researchers at Stanford University showed that student performance on essays, even more than their standardised test scores, reflects the advantages of family income.
And another study earlier this year by Taylor Odle, an assistant professor of educational policy studies at the University of Wisconsin at Madison, and Preston Magouirk, a former research manager at the Common App – the unified online college admissions platform – found that the essay was the chief place at which US students drop out of the application process. Those dropouts from the Common App disproportionately involve racial minorities, Professor Odle and Dr Magouirk found.
Such findings suggest that colleges and universities should either stop using essays in admissions or greatly reduce their reliance on them, said James Murphy, the deputy director of higher education policy at the advocacy group Education Reform Now.
“I would call it a very bad solution to a problem that barely exists,” Dr Murphy said of the Penn-led study’s encouragement of AI-rated essays. “They’re trying to come up with a cheap and efficient way to do something that I would argue simply isn’t worth doing.” Better strategies for improving equity in admissions, he said, include reducing acceptances from private high schools, ending legacy preferences, and reducing sports-based advantages.
Professor Duckworth said her study identifies value in admissions essays but doesn’t definitively argue for their use. “We’re not pretending that this is a panacea for anything,” she said. All measures of college applicants – including grade-point averages, extracurricular activities, teacher recommendations and essays – have shortcomings that must be recognised and managed by admissions officers, she said.
“If you remove each element, one by one, because they are flawed,” Professor Duckworth said, “then I think you can guarantee that you’ll have nothing left, because there are no perfect elements of admissions.”
