A new report by Stanford University has revealed that the high cost of training Artificial Intelligence (AI) models is limiting the involvement of non-industry actors in the technology revolution.
According to the Artificial Intelligence Index Report 2024, while AI companies seldom reveal the expenses involved in training their models, it is widely believed that these costs run into millions of dollars and are rising.
The report cited openAI as an instance, where the company’s CEO, Sam Altman, said in 2023 lthat the training cost for GPT-4 was over $100 million.
- “This escalation in training expenses has effectively excluded universities, traditionally centers of AI research, from developing their own leading-edge foundation models,” the report stated.
This revelation comes as the Nigerian government launched its Large Language Model (LLM), which is expected to position the country as AI leader in Africa. According to the Minister of Communications, Innovation and Digital Economy, Dr. Bosun Tijani, the LLM will be trained in five low-resource languages and accented English to ensure stronger language representation in existing datasets for the development of AI solutions.
What tech spent on training AI
To illustrate the growing cost of training AI, the researchers at Stanford cited an example of the Transformer model, which introduced the architecture that underpins virtually every modern LLM, noting that it costs only around $900 to train. They, however, estimated that OpenAI’s GPT-4 costs $78 million while Google’s Gemini Ultra costs around $191 million.
- “AI Index estimates validate suspicions that in recent years model training costs have significantly increased.
- “For example, in 2017, the original Transformer model, which introduced the architecture that underpins virtually every modern LLM, cost around $900 to train.
- “RoBERTa Large, released in 2019, which achieved state-of-the-art results on many canonical comprehension benchmarks like SQuAD and GLUE, cost around $160,000 to train.
- “Fast-forward to 2023, and training costs for OpenAI’s GPT-4 and Google’s Gemini Ultra are estimated to be around $78 million and $191 million, respectively,” the report stated.
Data challenge
The report further noted that another challenge that may be encountered in launching into AI is data. According to the report, LLMs have been trained on meaningful percentages of all the data that has ever existed on the internet.
- “The growing data dependency of AI models has led to concerns that future generations of computer scientists will run out of data to further scale and improve their systems
- “Research from Epoch suggests that these concerns are somewhat warranted. Epoch researchers have generated historical and compute-based projections for when AI researchers might expect to run out of data
- “For instance, the researchers estimate that computer scientists could deplete the stock of high-quality language data by 2024, exhaust low-quality language data within two decades, and use up image data by the late 2030s to mid-2040s,” Stanford University stated in the report.
