By ITU News 23rd September 2022
As artificial intelligence (AI) steadily grows, so do concerns about its environmental footprint. Today’s emerging natural language processing (NLP) models, such as GPT-3 can consume as much energy as five cars, according to a 2019 study.
To reduce their environmental and climate impact, researchers in the United Arab Emirates are proposing a new development approach for these models that takes energy consumption into account at every stage, aiming to boost energy efficiency wherever possible.
Last April, Abu Dhabi’s Technology Innovation Institute (TII) launched NOOR, the largest Arabic-language NLP model to date.
NOOR – Arabic for “light” – is trained on 10 billion parameters including books, poetry, news, and technical information, reinforcing the model’s broad applicability, according to its creators.
NLP systems, among other subfields of AI and linguistics, allow computers to understand, interpret and manipulate human language, based on deep learning-based training. The more parameters used in development, the more capable the system becomes.
But balancing system capabilities with the energy costs and environmental impact presents major challenges for AI and NLP developers.
The burden of inferences
An exhaustive study by TII shows NOOR’s average carbon footprint and indicates the energy cost behind each of the Arabic NLP model’s processes, explains Ebtesam Almazrouei, Director of the Al-Cross Center Unit at TII.
Model inference – when the AI system uses what it learns during training to make a prediction – is the most resource-intensive, TII’s research reveals. “When I run this AI on my devices, there is a real inference on how much the cost will be,” Almazrouei noted in a recent AI for Good webinar, “the scale of the model is linearly correlated with its energy and carbon footprint.”
External factors related to the project also have a significant impact. One example is international air travel. Since NOOR was developed together with the French technology company LightOn, which specialises in extreme-scale foundation models, the team travelled between Paris and Abu Dhabi several times.
“We want to achieve similar state-of-the-art capabilities to the GPT-3 model,” Almazrouei said about the aspirations of the team she leads, adding: “NOOR will become the reference language model for Arabic.”
With English and Chinese accounting for most progress in NPL models to date, she emphasises the need to “empower” NPL use in other languages.
Future applications for NOOR could range from speech recognition to automated text synthesis and chatbot services.
Consumption efficiency
The data centres used in training models are also part of the quest to make advanced AI energy efficient. Their power usage effectiveness ratio, or PUE, represents their total energy use divided by the amount used by their computing equipment.
While the world’s largest data centres have a PUE of 1.57, Google says its centres have achieved an average PUE of 1.1 by using hardware materials that emit less heat and therefore require less energy for cooling.
Energy efficiency also varies by country and region, reflecting the efficiency of different energy grids. The carbon intensity of the local electricity mix “significantly impacts the final footprint,” according to Almazrouei.
Choosing a location with green energy sources, therefore, lower’s a project’s environmental impact and ecological costs.
While some countries and regions are trying to reduce emissions from their data centres, the task is complex. One European Commission study for example notes the difficulty of making data centres climate-neutral and highly energy-efficient by 2030, as per the European Digital Strategy.
Estimates say the average consumption of data centres on the continent will grow by 28 per cent by the end of the decade.
Carbon footprint as a performance metric
Eco-efficient criteria must be considered in every decision from the outset, based on the needs and characteristics of each model, with AI developers determining “the best trade-off I can make to still have a good accuracy for my client.”
Up to now, environmental footprint reporting for AI has focused largely on the computational resources used in training new models. In the TII team’s view, this must extend to all phases of model development and use, with carbon footprints becoming a metric in assessing model quality and performance.
Source: https://www.itu.int