The report, Can We Trust AI on Climate?, was presented at the COP31 climate conference in Antalya, Türkiye, by the United Nations (UN) Educational, Scientific and Cultural Organisation (UNESCO) and University College London.
Researchers tested up to 33 leading AI models across 13 languages to assess factual knowledge, scientific reasoning, responses to disagreement and performance across different languages.
UNESCO Report Lead Author Leona Verdadero said AI chatbots often delivered responses that sounded authoritative, even when the information was inaccurate. “Many of us interact with these AI models and chatbots on a daily basis, seeking information on a wide variety of topics,” Verdadero said.
“A lot of times these models, when they respond, sound really authoritative and very smart. But as we’ve seen with our experiments, they don’t always sound right.”
The report found that AI systems tended towards sycophancy, described as a built-in inclination to agree with users even when users presented false information.
Researchers tested 15 models by challenging answers the systems had initially answered correctly. The challenges progressed from simple disagreement to claims of expertise and fabricated evidence.
Every model eventually changed a correct answer, while some could be persuaded to endorse false claims, including that carbon dioxide was not the main driver of climate change.
About eight in 10 responses that changed did so after the first expression of disagreement from a user. “If a user says, ‘I actually don’t think that is right,’ what is very fascinating is that we found in eight out of 10 of these cases, when a user pushes back, the AI model can actually reverse its answer to be more agreeable with the user,” Verdadero said.
“AI could just abandon a scientific fact to be more agreeable to the user.”
The strongest models performed well when answering textbook-style climate questions, achieving accuracy rates of between 88% and 96%.
However, performance deteriorated when questions required more complex reasoning.
When asked to determine whether a piece of evidence supported or contradicted a scientific claim, the strongest systems recorded accuracy rates of between 71% and 77%.
Models performed worse when asked whether two climate policies implemented together would produce greater, smaller or equivalent effects compared with their individual outcomes. The models achieved an average accuracy rate of 33.8%, which the report said was no better than random guessing.
“What we are saying is that the science and the experts should be trusted more,” Verdadero said. “The AI models are not there yet.”
The language used to ask questions also affected the accuracy of responses.
The models achieved an average accuracy rate of 88.6% when tested in English, with similar results recorded in Spanish, French and Portuguese.
Accuracy declined to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili.
“Imagine you and I are using the same model, asking the exact same question, but in a different language,” Verdadero said. “Based on the language, we can have totally different responses.”
Verdadero said communities speaking languages in which the models performed poorly were primarily located in South Asia and East Africa, regions already vulnerable to the effects of climate change.
The report called on technology companies to invest more in local datasets and multilingual AI development to prevent speakers of underrepresented languages from being left behind.
The research forms part of UNESCO’s contribution to the Global Initiative for Information Integrity on Climate Change, which UNESCO co-chairs with the UN and Brazil.
“The fight against climate change needs to be grounded in reliable evidence and scientific evidence,” Verdadero said. “A lot of this evidence informs policies today and also feeds into the COP negotiations.”
Verdadero said AI should be treated as one source of information rather than a primary authority. “These AI models should be used as one type of information source, but they should not be your primary one,” Verdadero said.
“Fact-checking and making sure the sources are reliable is important. Trusting evidence from experts, climate scientists and researchers should not be replaced.”
–UN/ChannelAfrica–
Image: Reuters