As funding bodies explore using artificial intelligence to assess grant applications, a study has raised concerns that these tools could favour established research over more novel ideas. AI may offer speed and efficiency, but its assessments could also influence which projects receive support.
In research published in Scientometrics in August, Diogo Machado asked OpenAI’s GPT-4o to compare 139 genuine proposals submitted to the Austrian Science Fund in 2021. The model assessed proposals in pairs, choosing which one should receive funding. Those with the most selections were ranked highest.
The proposals also included human reviewers’ evaluations of novelty, scientific quality, team capacity and feasibility, along with the final funding decisions. GPT-4o’s choices matched the human reviewers’ decisions in 60% of cases. However, Machado found that the model was systematically less likely than the reviewers to select proposals rated highly for novelty.
Machado, who leads the data-science team at science-policy consultancy Technopolis, said the results point to a risk that AI-assisted assessment could steer funding towards research grounded in established ideas and overlook potential breakthroughs.
He suspects that large language models may struggle to identify novelty because they learn from recurring patterns in large datasets. Mainstream ideas appear more often in those data, while genuinely new contributions are less common, leaving fewer patterns for a model to draw on when judging their value. The study did not test why the bias occurred, however.
The analysis also had a limitation: GPT-4o was asked simply which proposal should be funded, rather than being given the specific criteria used by human reviewers. But applying a novelty criterion would not necessarily solve the problem. Susan Guthrie, director of science and emerging technology at RAND Europe, noted that novelty is subjective and difficult to standardize.
Several major funders are examining how AI might be used in grant assessment. UK Research and Innovation said last month it was exploring AI-assisted assessment through a controlled trial that will compare AI-generated evaluations with past funding decisions. Wellcome, meanwhile, prohibits reviewers from entering grant applications into generative AI tools over privacy concerns, while allowing their use for side tasks such as checking whether similar research has already been conducted.
Machado said understanding the biases AI tools may introduce could help funders develop safeguards.
Sources: Nature.