View source for Neural scaling law

{{Short description|Statistical law in machine learning}}
[[File:Performance_of_AI_models_on_various_benchmarks_from_1998_to_2024.png|thumb|Performance of AI models on various benchmarks from 1998 to 2024.]]
In [[machine learning]], a '''neural scaling law''' is an empirical [[scaling law]] that describes how [[Neural network (machine learning)|neural network]] performance changes as key factors are scaled up or down. These factors typically include the number of parameters, [[Training, validation, and test data sets|training dataset]] size,<ref>{{cite journal |last1=Bahri |first1=Yasaman |last2=Dyer |first2=Ethan |last3=Kaplan |first3=Jared |last4=Lee |first4=Jaehoon |last5=Sharma |first5=Utkarsh |date=2024 |title=Explaining neural scaling laws |journal=[[Proceedings of the National Academy of Sciences of the United States of America|Proceedings of the National Academy of Sciences]] |volume=121 |issue=27 |pages=e2311878121 |arxiv=2102.06701 |doi=10.1073/pnas.2311878121 |doi-access=free|pmid=38913889 |pmc=11228526 |bibcode=2024PNAS..12111878B }}</ref><ref name=":4">{{Cite arXiv |eprint=1712.00409 |class=cs.LG |first1=Joel |last1=Hestness |first2=Sharan |last2=Narang |title=Deep Learning Scaling is Predictable, Empirically |date=2017-12-01 |last3=Ardalani |first3=Newsha |last4=Diamos |first4=Gregory |last5=Jun |first5=Heewoo |last6=Kianinejad |first6=Hassan |last7=Patwary |first7=Md Mostofa Ali |last8=Yang |first8=Yang |last9=Zhou |first9=Yanqi}}</ref> and training cost.

== Introduction ==
In general, a [[deep learning]] model can be characterized by four parameters: model size, training dataset size, training cost, and the post-training error rate (e.g., the test set error rate). Each of these variables can be defined as a real number, usually written as <math>N, D, C, L</math> (respectively: parameter count, dataset size, computing cost, and [[Loss function|loss]]).

A neural scaling law is a theoretical or empirical [[Empirical statistical laws|statistical law]] between these parameters. There are also other parameters with other scaling laws.

=== Size of the model ===
In most cases, the model's size is simply the number of parameters. However, one complication arises with the use of sparse models, such as [[Mixture of experts|mixture-of-expert models]].<ref>{{Cite journal |last1=Rajbhandari |first1=Samyam |last2=Li |first2=Conglong |last3=Yao |first3=Zhewei |last4=Zhang |first4=Minjia |last5=Aminabadi |first5=Reza Yazdani |last6=Awan |first6=Ammar Ahmad |last7=Rasley |first7=Jeff |last8=He |first8=Yuxiong |date=2022-06-28 |title=DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale |url=https://proceedings.mlr.press/v162/rajbhandari22a.html |journal=Proceedings of the 39th International Conference on Machine Learning |language=en |publisher=PMLR |pages=18332–18346|arxiv=2201.05596 }}</ref> With sparse models, during inference, only a fraction of their parameters are used. In comparison, most other kinds of neural networks, such as [[Transformer (deep learning architecture)|transformer]] models, always use all their parameters during inference.

=== Size of the training dataset ===
The size of the training dataset is usually quantified by the number of data points within it. Larger training datasets are typically preferred, as they provide a richer and more diverse source of information from which the model can learn. This can lead to improved generalization performance when the model is applied to new, unseen data.<ref name="goodfellow">Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.</ref> However, increasing the size of the training dataset also increases the computational resources and time required for model training.

With the "pretrain, then finetune" method used for most [[large language model]]s, there are two kinds of training dataset: the ''pretraining'' dataset and the ''finetuning'' dataset. Their sizes have different effects on model performance. Generally, the finetuning dataset is less than 1% the size of pretraining dataset.<ref name=":2">{{Cite arXiv|last1=Zhou |first1=Chunting |last2=Liu |first2=Pengfei |last3=Xu |first3=Puxin |last4=Iyer |first4=Srini |last5=Sun |first5=Jiao |last6=Mao |first6=Yuning |last7=Ma |first7=Xuezhe |last8=Efrat |first8=Avia |last9=Yu |first9=Ping |last10=Yu |first10=Lili |last11=Zhang |first11=Susan |last12=Ghosh |first12=Gargi |last13=Lewis |first13=Mike |last14=Zettlemoyer |first14=Luke |last15=Levy |first15=Omer |date=2023-05-01 |title=LIMA: Less Is More for Alignment |class=cs.CL |eprint=2305.11206 }}</ref>

In some cases, a small amount of high quality data suffices for finetuning, and more data does not necessarily improve performance.<ref name=":2" />

=== Cost of training ===
[[File:Amortized_hardware_and_energy_cost_to_train_frontier_AI_models_over_time.png|thumb|Amortized hardware and energy cost to train frontier AI models over time.]]
Training cost is typically measured in terms of time (how long it takes to train the model) and computational resources (how much processing power and memory are required). It is important to note that the cost of training can be significantly reduced with efficient training algorithms, optimized software libraries, and [[parallel computing]] on specialized hardware such as [[Graphics processing unit|GPUs]] or [[Tensor processing unit|TPUs]].

The cost of training a neural network model is a function of several factors, including model size, training dataset size, the training algorithm [[Computational complexity theory|complexity]], and the computational resources available.<ref name="goodfellow" /> In particular, doubling the training dataset size does not necessarily double the cost of training, because one may train the model for several times over the same dataset (each being an "[[Glossary of artificial intelligence#epoch|epoch]]").

=== Performance ===
[[File:MMLU_performance_vs_AI_scale.png|thumb|MMLU performance vs AI scale as a sigmoid]][[File:Big_Bench_Hard_performance_vs_AI_scale.png|thumb|BIG-Bench (hard)<ref>{{cite web |title=google/BIG-bench |date=2024-09-24 |url=https://github.com/google/BIG-bench/tree/main |access-date=2024-09-25 |publisher=Google}}</ref> performance vs AI scale as a sigmoid.]]{{See also|List of language model benchmarks}}
[[File:Densing_Law_of_LLMs,_Figure_2.png|thumb|Performance on several benchmarks vs negative log-likelihood loss <math>L</math> (with <math>L_0</math> removed), fitted as sigmoid functions.<ref name=":7" />{{Pg|location=Figure 2}}]]
The performance of a neural network model is evaluated based on its ability to accurately predict the output given some input data. Common metrics for evaluating model performance include:<ref name="goodfellow" />

* Negative [[Likelihood function|log-likelihood]] per token (logarithm of [[perplexity]]) for [[language model]]ing;
* [[Accuracy and precision|Accuracy]], [[Precision and recall|precision, recall]], and [[F1 score]] for [[classification]] tasks;
* [[Mean squared error]] (MSE) or [[mean absolute error]] (MAE) for [[Regression analysis|regression]] tasks;
* [[Elo rating system|Elo rating]] in a competition against other models, such as [[Computer chess|gameplay]]<ref name=":6">{{cite arXiv | eprint=2104.03113 | last1=Jones | first1=Andy L. | title=Scaling Scaling Laws with Board Games | date=2021 | class=cs.LG }}</ref> or [[Reinforcement learning from human feedback|preference by a human judge]].<ref>[https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard LMSYS Chatbot leaderboard]</ref>

Performance can be improved by using more data, larger models, different training algorithms, [[Regularization (mathematics)|regularizing]] the model to prevent [[overfitting]], and early stopping using a validation set.

When the performance is a number bounded within the range of <math>[0, 1]</math>, such as accuracy, precision, etc., it often scales as a [[sigmoid function]] of cost, as seen in the figures.

== Examples ==

=== (Hestness, Narang, et al, 2017) ===
The 2017 paper<ref name=":4" /> is a common reference point for neural scaling laws fitted by statistical analysis on experimental data. Previous works before the 2000s, as cited in the paper, were either theoretical or orders of magnitude smaller in scale. Whereas previous works generally found the scaling exponent to scale like <math>L \propto D^{-\alpha} </math>, with <math>\alpha \in \{0.5, 1, 2\}  </math>, the paper found that <math>\alpha \in [0.07, 0.35]</math>.

Of the factors they varied, only task can change the exponent <math>\alpha</math>. Changing the architecture optimizers, regularizers, and loss functions, would only change the proportionality factor, not the exponent. For example, for the same task, one architecture might have <math>L = 1000 D^{-0.3} </math> while another might have <math>L = 500 D^{-0.3}</math>. They also found that for a given architecture, the number of parameters necessary to reach lowest levels of loss, given a fixed dataset size, grows like <math>N \propto D^{\beta}</math> for another exponent <math>\beta</math>.

They studied machine translation with LSTM (<math>\alpha \sim 0.13  </math>), generative language modelling with LSTM (<math>\alpha \in [0.06, 0.09], \beta \approx 0.7</math>), ImageNet classification with ResNet (<math>\alpha \in [0.3, 0.5], \beta \approx 0.6</math>), and speech recognition with two hybrid (LSTMs complemented by either CNNs or an attention decoder) architectures (<math>\alpha \approx 0.3</math>).

=== (Henighan, Kaplan, et al, 2020) ===
A 2020 analysis <ref name=":0">{{Cite book |last1=Henighan |first1=Tom |title=Scaling Laws for Autoregressive Generative Modeling |last2=Kaplan |first2=Jared |last3=Katz |first3=Mor |last4=Chen |first4=Mark |last5=Hesse |first5=Christopher |last6=Jackson |first6=Jacob |last7=Heewoo |first7=Jun |last8=Brown |first8=Tom B. |last9=Dhariwal |first9=Prafulla |date=2020-10-27 |arxiv=2010.14701 |oclc=1228442047 |last10=Mann |first10=Chris |last11=Radford |first11=Alec |last12=Ramesh |first12=Aditya |last13=Ryder |first13=Nick |last14=Ziegler |first14=Daniel M. |last15=Schulman |first15=John |last16=Gray |first16=Scott |last17=Hallacy |first17=Chris |last18=Amodei |first18=Dario |last19=McCandlish |first19=Sam}}</ref> studied statistical relations between <math>C, N, D, L</math> over a wide range of values and found similar scaling laws, over the range of <math>N \in [10^3, 10^9]</math>, <math>C\in [10^{12}, 10^{21}]</math>, and over multiple modalities (text, video, image, text to image, etc.).<ref name=":0" />

In particular, the scaling laws it found are (Table 1 of <ref name=":0" />):

* For each modality, they fixed one of the two <math>C, N</math>, and varying the other one (<math>D</math> is varied along using <math>D = C/6N</math>), the achievable test loss satisfies<math display="block">L = L_0 + \left( \frac{x_0}{x}\right)^\alpha</math>where <math>x</math> is the varied variable, and <math>L_0, x_0, \alpha</math> are parameters to be found by statistical fitting. The parameter <math>\alpha</math> is the most important one. 
** When <math>N</math> is the varied variable, <math>\alpha</math> ranges from <math>0.037 </math> to <math>0.24</math> depending on the model modality. This corresponds to the <math>\alpha = 0.34 </math> from the Chinchilla scaling paper.
** When <math>C</math> is the varied variable, <math>\alpha</math> ranges from <math>0.048 </math> to <math>0.19</math> depending on the model modality. This corresponds to the <math>\beta = 0.28 </math> from the Chinchilla scaling paper.
* Given fixed computing budget, optimal model parameter count is consistently around<math display="block">N_{opt}(C) =\left(\frac{C}{5\times 10^{-12}\text{petaFLOP-day}}\right)^{0.7} = 9.0\times 10^{-7} C^{0.7}</math>The parameter <math>9.0 \times 10^{-7}</math> varies by a factor of up to 10 for different modalities. The exponent parameter <math>0.7</math> varies from <math>0.64</math> to <math>0.75</math> for different modalities. This exponent corresponds to the <math>\approx 0.5</math> from the Chinchilla scaling paper.
* It's "strongly suggested" (but not statistically checked) that <math>D_{opt}(C) \propto N_{opt}(C)^{0.4} \propto C^{0.28}</math>. This exponent corresponds to the <math>\approx 0.5</math> from the Chinchilla scaling paper.

The scaling law of <math>L = L_0 + (C_0/C)^{0.048}</math> was confirmed during the training of [[GPT-3]] (Figure 3.1 <ref>{{Cite arXiv |last1=Brown |first1=Tom B. |last2=Mann |first2=Benjamin |last3=Ryder |first3=Nick |last4=Subbiah |first4=Melanie |last5=Kaplan |first5=J. |last6=Dhariwal |first6=Prafulla |last7=Neelakantan |first7=Arvind |last8=Shyam |first8=Pranav |last9=Sastry |first9=Girish |last10=Askell |first10=Amanda |last11=Agarwal |first11=Sandhini |last12=Herbert-Voss |first12=Ariel |last13=Krueger |first13=Gretchen |last14=Henighan |first14=T. |last15=Child |first15=Rewon |date=2020-05-28 |title=Language Models are Few-Shot Learners |class=cs.CL |eprint=2005.14165 }}</ref>).

=== Chinchilla scaling (Hoffmann, et al, 2022) ===
[[File:Optimal_ratio_of_training_tokens_to_model_parameters_for_Chinchilla_scaling_law.png|thumb|Optimal ratio of training tokens to model parameters for Chinchilla scaling law. It shows that in general, "Chinchilla optimal" scaling is <math>D = 20 N</math>, and is significantly different from (Hoffmann et al., 2022). Data analysis by Epoch AI.<ref>{{Cite web |last=Besiroglu |first=Tamay |date=2024-04-17 |title=Chinchilla Scaling: A Replication Attempt |url=https://epochai.org/blog/chinchilla-scaling-a-replication-attempt |access-date=2024-09-24 |website=Epoch AI |language=en}}</ref>]]
One particular scaling law ("[[Chinchilla AI|Chinchilla scaling]]") states that, for a [[large language model]] (LLM) autoregressively trained for one epoch, with a cosine [[learning rate]] schedule, we have:<ref>{{Cite arXiv |last1=Hoffmann |first1=Jordan |last2=Borgeaud |first2=Sebastian |last3=Mensch |first3=Arthur |last4=Buchatskaya |first4=Elena |last5=Cai |first5=Trevor |last6=Rutherford |first6=Eliza |last7=Casas |first7=Diego de Las |last8=Hendricks |first8=Lisa Anne |last9=Welbl |first9=Johannes |last10=Clark |first10=Aidan |last11=Hennigan |first11=Tom |last12=Noland |first12=Eric |last13=Millican |first13=Katie |last14=Driessche |first14=George van den |last15=Damoc |first15=Bogdan |date=2022-03-29 |title=Training Compute-Optimal Large Language Models |class=cs.CL |eprint=2203.15556 }}</ref><math display="block">\begin{cases}
C = C_0 ND\\
L = \frac{A}{N^\alpha} + \frac{B}{D^{\beta}} + L_0
\end{cases}</math>where the variables are

* <math>C</math> is the cost of training the model, in [[FLOPS]].
* <math>N</math> is the number of parameters in the model.
* <math>D</math> is the number of tokens in the training set.
* <math>L</math> is the average negative log-likelihood loss per token ([[Nat (unit)|nats]]/token), achieved by the trained LLM on the test dataset.
** <math>L_0</math> represents the loss of an ideal generative process on the test data
** <math>\frac{A}{N^\alpha}</math> captures the fact that a Transformer language model with <math>N</math> parameters underperforms the ideal generative process
** <math>\frac{B}{D^\beta}</math> captures the fact that the model trained on <math>D</math> tokens underperforms the ideal generative process

and the statistical parameters are

* <math> C_0 = 6</math>, meaning that it costs 6 FLOPs per parameter to train on one token. This is estimated by Kaplan et al.<ref name="kaplan-scaling">{{Cite journal |last1=Kaplan |first1=Jared |last2=McCandlish |first2=Sam |last3=Henighan |first3=Tom |last4=Brown |first4=Tom B. |last5=Chess |first5=Benjamin |last6=Child |first6=Rewon |last7=Gray |first7=Scott |last8=Radford |first8=Alec |last9=Wu |first9=Jeffrey |last10=Amodei |first10=Dario |title=Scaling Laws for Neural Language Models |journal=CoRR |volume=abs/2001.08361 |year=2020 |arxiv=2001.08361}}</ref> Note that training cost is much higher than inference cost, as training entails both forward and [[Backpropagation|backward passes]], whereas inference costs 1 to 2 FLOPs per parameter to infer on one token.
* <math>\alpha = 0.34, \beta = 0.28, A = 406.4, B = 410.7, L_0 = 1.69</math>.
Although Besiroglu et al.<ref>{{cite arXiv |last1=Besiroglu |first1=Tamay |title=Chinchilla Scaling: A replication attempt |date=2024-04-15 |eprint=2404.10102 |last2=Erdil |first2=Ege |last3=Barnett |first3=Matthew |last4=You |first4=Josh|class=cs.AI }}</ref> claims that the statistical estimation is slightly off, and should be <math>\alpha = 0.35, \beta = 0.37, A = 482.01, B = 2085.43, L_0 = 1.82</math>.

The statistical laws were fitted over experimental data with <math>N\in [7\times 10^7, 1.6 \times 10^{10}], D \in [5\times 10^9, 5\times 10^{11}], C \in [10^{18}, 10^{24}]</math>.

Since there are 4 variables related by 2 equations, imposing 1 additional constraint and 1 additional [[Mathematical optimization|optimization objective]] allows us to solve for all four variables. In particular, for any fixed <math>C</math>, we can uniquely solve for all 4 variables that minimizes <math>L</math>. This provides us with the optimal <math>D_{opt}(C), N_{opt}(C)</math> for any fixed <math>C</math>:<math display="block">N_{o p t}(C)=G\left(\frac{C}{6}\right)^a, \quad D_{o p t}(C)=G^{-1}\left(\frac{C}{6}\right)^b, \quad \text { where } \quad G=\left(\frac{\alpha A}{\beta B}\right)^{\frac{1}{\alpha+\beta}}, \quad a=\frac{\beta}{\alpha+\beta} \text {, and } b=\frac{\alpha}{\alpha+\beta} \text {. }</math>Plugging in the numerical values, we obtain the "Chinchilla efficient" model size and training dataset size, as well as the test loss achievable:<math display="block">\begin{cases}
N_{opt}(C) = 0.6 \; C^{0.45} \\
D_{opt}(C) = 0.3 \; C^{0.55} \\
L_{opt}(C) = 1070 \; C^{-0.154} + 1.7
\end{cases}</math>Similarly, we may find the optimal training dataset size and training compute budget for any fixed model parameter size, and so on.

There are other estimates for "Chinchilla efficient" model size and training dataset size. The above is based on a statistical model of <math>L = \frac{A}{N^\alpha} + \frac{B}{D^{\beta}} + L_0</math>. One can also directly fit a statistical law for <math>D_{opt}(C), N_{opt}(C)</math> without going through the detour, for which one obtains:<math display="block">\begin{cases}
N_{opt}(C) = 0.1 \; C^{0.5}\\
D_{opt}(C) = 1.7 \; C^{0.5}
\end{cases}</math>or as tabulated:
{| class="wikitable"
|+
!<math>N_{opt}(C)</math>
!<math>C</math> / FLOP
!<math>C</math> / FLOPs of training Gopher
!<math>D_{opt}(C)</math>
|-
|400 million
|1.92e+19
|1/29968
|8.0 billion
|-
|1 billion
|1.21e+20
|1/5706
|20.2 billion
|-
|10 billion
|1.23e+22
|1/2819
|205.1 billion
|-
|67 billion
|5.76e+23
|1
|1.5 trillion
|-
|175 billion
|3.85e+24
|6.7
|3.7 trillion
|-
|280 billion
|9.90e+24
|17.2
|5.9 trillion
|-
|520 billion
|3.43e+25
|59.5
|11.0 trillion
|-
|1 trillion
|1.27e+26
|221.3
|21.2 trillion
|-
|10 trillion
|1.30e+28
|22515.9
|216.2 trillion
|}

==== Discrepancy ====
The Chinchilla scaling law analysis for training [[Transformer (machine learning model)|transformer]] language models suggests that for a given training compute budget (<math>C</math>), to achieve the minimal pretraining loss for that budget, the number of model parameters (<math>N</math>) and the number of training tokens (<math>D</math>) should be scaled in equal proportions, <math>N_{opt}(C) \propto C^{0.5}, D_{opt}(C) \propto C^{0.5}</math>. 
This conclusion differs from analysis conducted by Kaplan et al.,<ref name="kaplan-scaling" /> which found that <math>N</math> should be increased more quickly than <math>D</math>, <math>N_{opt}(C) \propto C^{0.73}, D_{opt}(C) \propto C^{0.27}</math>.

This discrepancy can primarily be attributed to the two studies using different methods for measuring model size. Kaplan et al.:<ref>{{Citation |last1=Pearce |first1=Tim |title=Reconciling Kaplan and Chinchilla Scaling Laws |date=2024 |arxiv=2406.12907 |last2=Song |first2=Jinyeop}}</ref>

* did not count the parameters in the token embedding layer, which when analyzed at smaller model sizes leads to biased coefficients; 
* studied smaller models than the Chinchilla group, magnifying the effect;
* assumed that <math>L_\infty = 0</math>.

Secondary effects also arise due to differences in hyperparameter tuning and learning rate schedules. Kaplan et al.:<ref>{{Citation |last1=Porian |first1=Tomer |title=Resolving Discrepancies in Compute-Optimal Scaling of Language Models |date=2024-07-25 |arxiv=2406.19146 |last2=Wortsman |first2=Mitchell |last3=Jitsev |first3=Jenia |last4=Schmidt |first4=Ludwig |last5=Carmon |first5=Yair}}</ref>

* used a warmup schedule that was too long for smaller models, making them appear less efficient;
* did not fully tuning optimization hyperparameters.

==== Beyond Chinchilla scaling ====
As Chinchilla scaling has been the reference point for many large-scaling training runs, there had been a concurrent effort to go "beyond Chinchilla scaling", meaning to modify some of the training pipeline in order to obtain the same loss with less effort, or deliberately train for longer than what is "Chinchilla optimal".

Usually, the goal is to make the scaling law exponent larger, which means the same loss can be trained for much less compute. For instance, filtering data can make the scaling law exponent larger.<ref>{{cite arXiv |last1=Sorscher |first1=Ben |title=Beyond neural scaling laws: beating power law scaling via data pruning |date=2023-04-21 |eprint=2206.14486 |last2=Geirhos |first2=Robert |last3=Shekhar |first3=Shashank |last4=Ganguli |first4=Surya |last5=Morcos |first5=Ari S.|class=cs.LG }}</ref>

Another strand of research studies how to deal with limited data, as according to Chinchilla scaling laws, the training dataset size for the largest language models already approaches what is available on the internet.<ref>{{cite arXiv |last1=Tay |first1=Yi |title=Transcending Scaling Laws with 0.1% Extra Compute |date=2022-11-16 |eprint=2210.11399 |last2=Wei |first2=Jason |last3=Chung |first3=Hyung Won |last4=Tran |first4=Vinh Q. |last5=So |first5=David R. |last6=Shakeri |first6=Siamak |last7=Garcia |first7=Xavier |last8=Zheng |first8=Huaixiu Steven |last9=Rao |first9=Jinfeng|class=cs.CL }}</ref> found that augmenting the dataset with a mix of "denoising objectives" constructed from the dataset improves performance.<ref>{{Cite journal |last1=Muennighoff |first1=Niklas |last2=Rush |first2=Alexander |last3=Barak |first3=Boaz |last4=Le Scao |first4=Teven |last5=Tazi |first5=Nouamane |last6=Piktus |first6=Aleksandra |last7=Pyysalo |first7=Sampo |last8=Wolf |first8=Thomas |last9=Raffel |first9=Colin A. |date=2023-12-15 |title=Scaling Data-Constrained Language Models |url=https://proceedings.neurips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |language=en |volume=36 |pages=50358–50376|arxiv=2305.16264 }}</ref> studies optimal scaling when all available data is already exhausted (such as in rare languages), so one must train multiple epoches over the same dataset (whereas Chinchilla scaling requires only one epoch). The Phi series of small language models were trained on textbook-like data generated by large language models, for which data is only limited by amount of compute available.<ref>{{cite arXiv |last1=Li |first1=Yuanzhi |title=Textbooks Are All You Need II: phi-1.5 technical report |date=2023-09-11 |eprint=2309.05463 |last2=Bubeck |first2=Sébastien |last3=Eldan |first3=Ronen |last4=Del Giorno |first4=Allie |last5=Gunasekar |first5=Suriya |last6=Lee |first6=Yin Tat|class=cs.CL }}</ref>

Chinchilla optimality was defined as "optimal for training compute", whereas in actual production-quality models, there will be a lot of inference after training is complete. "Overtraining" during training means better performance during inference.<ref>{{cite arXiv |last1=Sardana |first1=Nikhil |title=Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws |date=2023-12-31 |eprint=2401.00448 |last2=Frankle |first2=Jonathan|class=cs.LG }}</ref> [[LLaMA]] models were overtrained for this reason. Subsequent studies discovered scaling laws in the overtraining regime, for dataset sizes up to 32x more than Chinchilla-optimal.<ref>{{cite arXiv |last1=Gadre |first1=Samir Yitzhak |title=Language models scale reliably with over-training and on downstream tasks |date=2024-03-13 |eprint=2403.08540 |last2=Smyrnis |first2=Georgios |last3=Shankar |first3=Vaishaal |last4=Gururangan |first4=Suchin |last5=Wortsman |first5=Mitchell |last6=Shao |first6=Rulin |last7=Mercat |first7=Jean |last8=Fang |first8=Alex |last9=Li |first9=Jeffrey|class=cs.CL }}</ref>

=== Broken neural scaling laws (BNSL) ===

A 2022 analysis<ref name=":1">{{cite arXiv | eprint=2210.14891 | last1=Caballero | first1=Ethan | last2=Gupta | first2=Kshitij | last3=Rish | first3=Irina | last4=Krueger | first4=David | title=Broken Neural Scaling Laws | date=2022 | class=cs.LG }}</ref> found that many scaling behaviors of artificial neural networks follow a [[Power law#Smoothly broken power law|smoothly broken power law]] functional form:

<math>y =  a + \bigg(bx^{-c_0}\bigg) \prod_{i=1}^n \left(1 + \left(\frac{x}{d_i}\right)^{1/f_i}\right)^{-c_i * f_i}</math>

in which <math>x</math> refers to the quantity being scaled (i.e. <math>C</math>, <math>N</math>, <math>D</math>, number of training steps, number of inference steps, or model input size) and <math>y</math> refers to the ''downstream'' (or upstream) performance evaluation metric of interest (e.g. prediction error, [[cross entropy]], calibration error, [[Receiver operating characteristic|AUROC]], [[BLEU|BLEU score percentage]], [[F1 Score|F1 score]], reward, [[Elo rating system|Elo rating]], solve rate, or [[Fréchet inception distance|FID]] score) in [[Zero-shot learning|zero-shot]], [[Prompt engineering|prompted]], or [[Fine-tuning (machine learning)|fine-tuned]] settings. The parameters <math>a, b, c_0, c_1 ... c_n, d_1 ...  d_n, f_1 ... f_n</math> are found by statistical fitting.

On a [[log–log plot]], when <math>f_i</math> is not too large and <math>a</math> is subtracted out from the y-axis, this functional form looks like a series of linear segments connected by arcs; the <math>n</math> transitions between the segments are called "breaks", hence the name ''broken neural scaling laws (BNSL)''.

The scenarios in which the scaling behaviors of artificial neural networks were found to follow this functional form include large-scale [[Computer vision|vision]], [[Natural language processing|language]], audio, video, [[Diffusion model|diffusion]], [[generative model]]ing, [[multimodal learning]], [[contrastive learning]], [[AI alignment]], AI capabilities, [[robotics]], out-of-distribution (OOD) generalization, continual learning, [[transfer learning]], [[Uncertainty quantification|uncertainty estimation]] / [[Calibration (statistics)|calibration]], [[Anomaly detection|out-of-distribution detection]], [[Adversarial machine learning|adversarial robustness]], [[Knowledge distillation|distillation]], sparsity, retrieval, quantization, [[Pruning (artificial neural network)|pruning]], [[Fairness (machine learning)|fairness]], molecules, computer programming/coding, math word problems, arithmetic, [[Large language model#Emergent abilities|emergent abilities]], [[double descent]], [[supervised learning]], [[Unsupervised learning|unsupervised]]/[[Self-supervised learning|self-supervised]] learning, and [[Deep reinforcement learning|reinforcement learning]] (single agent and [[Multi-agent reinforcement learning|multi-agent]]).

The architectures for which the scaling behaviors of artificial neural networks were found to follow this functional form include [[residual neural network]]s, [[Transformer (machine learning model)|transformers]], [[Multilayer perceptron|MLPs]], [[arxiv:2105.01601|MLP-mixers]], [[recurrent neural network]]s, [[convolutional neural network]]s, [[graph neural network]]s, [[U-Net|U-nets]], [[Transformer (machine learning model)#Encoder-decoder architecture|encoder-decoder]] (and [[Transformer (deep learning architecture)#encoder-only|encoder-only]]) (and decoder-only) models, [[Ensemble learning|ensembles]] (and non-ensembles), [[Mixture of experts|MoE]] (mixture of experts) (and non-MoE) models, and [[Pruning (artificial neural network)|sparse pruned]] (and non-sparse unpruned) models.

=== Inference scaling ===
[[File:Trading_Off_Compute_in_Training_and_Inference_in_the_game_of_Hex_for_AlphaZero_agents.png|thumb|The Elo rating of various AlphaZero agents trained to play the board game of Hex at varying train-time and test-time compute.]]{{Main|Reasoning language model}}
Other than scaling up training compute, one can also scale up inference compute (or "test-time compute"<ref>{{Citation |last1=Cobbe |first1=Karl |title=Training Verifiers to Solve Math Word Problems |date=2021-11-18 |arxiv=2110.14168 |last2=Kosaraju |first2=Vineet |last3=Bavarian |first3=Mohammad |last4=Chen |first4=Mark |last5=Jun |first5=Heewoo |last6=Kaiser |first6=Lukasz |last7=Plappert |first7=Matthias |last8=Tworek |first8=Jerry |last9=Hilton |first9=Jacob}}</ref>). As an example, the [[Elo rating system|Elo rating]] of [[AlphaGo]] improves steadily as it is allowed to spend more time on its [[Monte Carlo tree search|Monte Carlo Tree Search]] per play.<ref>{{Cite journal |last1=Silver |first1=David |last2=Huang |first2=Aja |last3=Maddison |first3=Chris J. |last4=Guez |first4=Arthur |last5=Sifre |first5=Laurent |last6=van den Driessche |first6=George |last7=Schrittwieser |first7=Julian |last8=Antonoglou |first8=Ioannis |last9=Panneershelvam |first9=Veda |last10=Lanctot |first10=Marc |last11=Dieleman |first11=Sander |last12=Grewe |first12=Dominik |last13=Nham |first13=John |last14=Kalchbrenner |first14=Nal |last15=Sutskever |first15=Ilya |date=January 2016 |title=Mastering the game of Go with deep neural networks and tree search |url=https://www.nature.com/articles/nature16961 |journal=Nature |language=en |volume=529 |issue=7587 |pages=484–489 |doi=10.1038/nature16961 |pmid=26819042 |bibcode=2016Natur.529..484S |issn=1476-4687}}</ref>{{Pg|location=Fig 4}} For [[AlphaGo Zero]], increasing Elo by 120 requires either 2x model size and training, or 2x test-time search.<ref name=":5">{{Cite AV media |url=https://www.youtube.com/watch?v=eaAonE58sLU |title=Parables on the Power of Planning in AI: From Poker to Diplomacy: Noam Brown (OpenAI) |date=2024-09-17 |last=Noam |first=Brown |type=Video |language=en |access-date=2024-09-24 |via=YouTube}}

Lecture at [[Paul G. Allen School of Computer Science & Engineering|Paul G. Allen School]] on Thursday, May 23, 2024, 3:30 pm</ref> Similarly, a language model for solving competition-level coding challenges, AlphaCode, consistently improved (log-linearly) in performance with more search time.<ref>{{Cite journal |last1=Li |first1=Yujia |last2=Choi |first2=David |last3=Chung |first3=Junyoung |last4=Kushman |first4=Nate |last5=Schrittwieser |first5=Julian |last6=Leblond |first6=Rémi |last7=Eccles |first7=Tom |last8=Keeling |first8=James |last9=Gimeno |first9=Felix |last10=Dal Lago |first10=Agustin |last11=Hubert |first11=Thomas |last12=Choy |first12=Peter |last13=de Masson d’Autume |first13=Cyprien |last14=Babuschkin |first14=Igor |last15=Chen |first15=Xinyun |date=2022-12-09 |title=Competition-level code generation with AlphaCode |url=https://www.science.org/doi/10.1126/science.abq1158 |journal=Science |language=en |volume=378 |issue=6624 |pages=1092–1097 |doi=10.1126/science.abq1158 |pmid=36480631 |arxiv=2203.07814 |bibcode=2022Sci...378.1092L |issn=0036-8075}}</ref>

For [[Hex (board game)|Hex]], 10x training-time compute trades for 15x test-time compute.<ref name=":6" /> For ''[[Libratus]]'' for heads up [[Betting in poker#No limit|no-limit]] [[Texas hold 'em]], and ''Cicero'' for ''[[Diplomacy (game)|Diplomacy]]'', and many other abstract games of partial information, inference-time searching improves performance at a similar tradeoff ratio, for up to 100,000x effective increase in training-time compute.<ref name=":5" />

In 2024, the [[OpenAI o1]] report documented that o1's performance consistently improved with both increased train-time compute and test-time compute, and gave numerous examples of test-time compute scaling in mathematics, scientific reasoning, and coding tasks.<ref>{{Cite web |last=Villalobos |first=Pablo |date=2023-07-28 |title=Trading Off Compute in Training and Inference |url=https://epochai.org/blog/trading-off-compute-in-training-and-inference |access-date=2024-09-24 |website=Epoch AI |language=en}}</ref><ref>{{cite web |title=Learning to Reason with LLMs |url=https://openai.com/index/learning-to-reason-with-llms/ |access-date=2024-09-16 |publisher=OpenAI}}</ref>

One method for scaling up test-time compute is '''process-based supervision''', where a model generates a step-by-step reasoning chain to answer a question, and another model (either human or AI) provides a reward score on some of the intermediate steps, not just the final answer. Process-based supervision can be scaled arbitrarily by using synthetic reward score without another model, for example, by running Monte Carlo rollouts and scoring each step in the reasoning according to how likely it leads to the right answer. Another method is by '''revision models''', which are models trained to solve a problem multiple times, each time revising the previous attempt.<ref>{{Citation |last1=Snell |first1=Charlie |title=Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters |date=2024-08-06 |arxiv=2408.03314 |last2=Lee |first2=Jaehoon |last3=Xu |first3=Kelvin |last4=Kumar |first4=Aviral}}</ref>

=== Other examples ===

==== Vision transformers ====
[[Vision transformer]]s, similar to language transformers, exhibit scaling laws. A 2022 research trained vision transformers, with parameter counts <math>N\in [5\times 10^6, 2\times 10^9]</math>, on image sets of sizes <math>D \in [3\times 10^{7}, 3\times 10^{9}]</math>, for computing <math>C\in [0.2, 10^4] </math> (in units of TPUv3-core-days).<ref>{{Cite journal |last1=Zhai |first1=Xiaohua |last2=Kolesnikov |first2=Alexander |last3=Houlsby |first3=Neil |last4=Beyer |first4=Lucas |date=2022 |title=Scaling Vision Transformers |url=https://openaccess.thecvf.com/content/CVPR2022/html/Zhai_Scaling_Vision_Transformers_CVPR_2022_paper.html |journal=CVPR |language=en |pages=12104–12113}}</ref>

After training the model, it is finetuned on [[ImageNet]] training set. Let <math>L</math> be the error probability of the finetuned model classifying ImageNet test set. They found <math>\min_{N, D} L = 0.09 + \frac {0.26}{(C + 0.01)^{0.35}}</math>.

==== Neural machine translation ====
Ghorbani, Behrooz et al.<ref name=":3">{{Cite arXiv |last1=Ghorbani |first1=Behrooz |last2=Firat |first2=Orhan |last3=Freitag |first3=Markus |last4=Bapna |first4=Ankur |last5=Krikun |first5=Maxim |last6=Garcia |first6=Xavier |last7=Chelba |first7=Ciprian |last8=Cherry |first8=Colin |date=2021-09-01 |title=Scaling Laws for Neural Machine Translation |class=cs.LG |eprint=2109.07740}}</ref> studied scaling laws for [[neural machine translation]] (specifically, English as source, and German as target) in encoder-decoder [[Transformer (machine learning model)|Transformer]] models, trained until convergence on the same datasets (thus they did not fit scaling laws for computing cost <math>C</math> or dataset size <math>D</math>). They varied <math>N \in [10^8, 3.5 \times 10^9]</math> They found three results:

* <math>L</math> is a scaling law function of <math>N_E, N_D</math>, where <math>N_E, N_D</math> are encoder and decoder parameter count. It is not simply a function of total parameter count <math>N = N_E + N_D</math>. The function has form <math>L\left(N_e, N_d\right)=\alpha\left(\frac{\bar{N}_e}{N_e}\right)^{p_e}\left(\frac{\bar{N}_d}{N_d}\right)^{p_d}+L_{\infty}</math>, where <math>\alpha, p_e, p_d, L_{\infty}, \bar N_e, \bar N_d</math> are fitted parameters. They found that <math>N_d/N \approx 0.55</math> minimizes loss if <math>N</math> is held fixed.
* <math>L</math> "saturates" (that is, it reaches <math>L_\infty</math>) for smaller models when the training and testing datasets are "source-natural" than "target-natural". A "source-natural" data point means a pair of English-German sentences, and the model is asked to translate the English sentence into German, and the English sentence is written by a natural English writer, while the German sentence is translated from the English sentence by a machine translator.<ref>{{Cite journal |last1=Chen |first1=Mia Xu |last2=Firat |first2=Orhan |last3=Bapna |first3=Ankur |last4=Johnson |first4=Melvin |last5=Macherey |first5=Wolfgang |last6=Foster |first6=George |last7=Jones |first7=Llion |last8=Schuster |first8=Mike |last9=Shazeer |first9=Noam |last10=Parmar |first10=Niki |last11=Vaswani |first11=Ashish |last12=Uszkoreit |first12=Jakob |last13=Kaiser |first13=Lukasz |last14=Chen |first14=Zhifeng |last15=Wu |first15=Yonghui |date=July 2018 |title=The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation |url=https://aclanthology.org/P18-1008 |journal=Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Melbourne, Australia |publisher=Association for Computational Linguistics |pages=76–86 |doi=10.18653/v1/P18-1008|arxiv=1804.09849 }}</ref> To construct the two kinds of datasets, the authors collected natural English and German sentences online, then used machine translation to generate their translations.
* As models grow larger, models trained on source-original datasets can achieve low loss but bad [[BLEU|BLEU score]]. In contrast, models trained on target-original datasets achieve low loss and good BLEU score in tandem (Figure 10, 11 <ref name=":3" />).

The authors hypothesize that source-natural datasets have uniform and dull target sentences, and so a model that is trained to predict the target sentences would quickly overfit.

<ref>{{Cite book |last1=Gordon |first1=Mitchell A |last2=Duh |first2=Kevin |last3=Kaplan |first3=Jared |title=Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing |chapter=Data and Parameter Scaling Laws for Neural Machine Translation |date=2021 |pages=5915–5922 |location=Stroudsburg, PA, USA |publisher=Association for Computational Linguistics |doi=10.18653/v1/2021.emnlp-main.478|doi-access=free }}</ref> trained Transformers for machine translations with sizes <math>N \in [4 \times 10^5 , 5.6 \times 10^7]</math> on dataset sizes <math>D \in [6\times 10^5, 6 \times 10^9]</math>. They found the Kaplan et al. (2020)<ref name="kaplan-scaling" /> scaling law applied to machine translation: <math>L(N, D)=\left[\left(\frac{N_C}{N}\right)^{\frac{\alpha_N}{\alpha_D}}+\frac{D_C}{D}\right]^{\alpha_D}</math>. They also found the BLEU score scaling as <math>BLEU \approx C e^{-kL}</math>.

==== Transfer learning ====
Hernandez, Danny et al.<ref>{{Cite arXiv |last1=Hernandez |first1=Danny |last2=Kaplan |first2=Jared |last3=Henighan |first3=Tom |last4=McCandlish |first4=Sam |date=2021-02-01 |title=Scaling Laws for Transfer |class=cs.LG |eprint=2102.01293}}</ref> studied scaling laws for [[transfer learning]] in language models. They trained a family of Transformers in three ways:

* pretraining on English, finetuning on Python
* pretraining on an equal mix of English and Python, finetuning on Python
* training on Python

The idea is that pretraining on English should help the model achieve low loss on a test set of Python text. Suppose the model has parameter count <math>N</math>, and after being finetuned on <math>D_F</math> Python tokens, it achieves some loss <math>L</math>. We say that its "transferred token count" is <math>D_T</math>, if another model with the same <math>N</math> achieves the same <math>L</math> after training on <math>D_F + D_T</math> Python tokens.

They found <math>D_T=1.9 e 4\left(D_F\right)^{.18}(N)^{.38}</math> for pretraining on English text, and <math>D_T=2.1 e 5\left(D_F\right)^{.096}(N)^{.38}</math> for pretraining on English and non-Python code.

==== Precision ====
Kumar et al.<ref>{{Citation |last1=Kumar |first1=Tanishq |title=Scaling Laws for Precision |date=2024-11-30 |url=https://arxiv.org/abs/2411.04330 |access-date=2025-02-17 |arxiv=2411.04330 |last2=Ankner |first2=Zachary |last3=Spector |first3=Benjamin F. |last4=Bordelon |first4=Blake |last5=Muennighoff |first5=Niklas |last6=Paul |first6=Mansheej |last7=Pehlevan |first7=Cengiz |last8=Ré |first8=Christopher |last9=Raghunathan |first9=Aditi}}</ref> study scaling laws for numerical precision in the training of language models. They train a family of language models with weights, activations, and KV cache in varying numerical precision in both integer and floating-point type to measure the effects on loss as a function of precision. For training, their scaling law accounts for lower precision by wrapping the effects of precision into an overall "effective parameter count" that governs loss scaling, using the parameterization <math>N \mapsto N_\text{eff}(P) = N(1-e^{-P/\gamma})</math>. This illustrates how training in lower precision degrades performance by reducing the true capacity of the model in a manner that varies exponentially with bits.

For inference, they find that extreme overtraining of language models past Chinchilla-optimality can lead to models being more sensitive to quantization, a standard technique for efficient deep learning. This is demonstrated by observing that the degradation in loss due to weight quantization increases as an approximate power law in the token/parameter ratio <math>D/N</math> seen during pretraining, so that models pretrained on extreme token budgets can perform worse in terms of validation loss than those trained on more modest token budgets if post-training quantization is applied. Other work examining the effects of overtraining include Sardana et al.<ref>{{Citation |last1=Sardana |first1=Nikhil |title=Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws |date=2024-07-18 |url=https://arxiv.org/abs/2401.00448 |access-date=2025-02-17 |arxiv=2401.00448 |last2=Portes |first2=Jacob |last3=Doubov |first3=Sasha |last4=Frankle |first4=Jonathan}}</ref> and Gadre et al.<ref>{{Citation |last1=Gadre |first1=Samir Yitzhak |title=Language models scale reliably with over-training and on downstream tasks |date=2024-06-14 |url=https://arxiv.org/abs/2403.08540 |access-date=2025-02-17 |arxiv=2403.08540 |last2=Smyrnis |first2=Georgios |last3=Shankar |first3=Vaishaal |last4=Gururangan |first4=Suchin |last5=Wortsman |first5=Mitchell |last6=Shao |first6=Rulin |last7=Mercat |first7=Jean |last8=Fang |first8=Alex |last9=Li |first9=Jeffrey}}</ref>

==== Densing laws ====
Xiao et al.<ref name=":7">{{Citation |last1=Xiao |first1=Chaojun |title=Densing Law of LLMs |date=2024-12-06 |url=https://arxiv.org/abs/2412.04315 |arxiv=2412.04315 |last2=Cai |first2=Jie |last3=Zhao |first3=Weilin |last4=Zeng |first4=Guoyang |last5=Lin |first5=Biyuan |last6=Zhou |first6=Jie |last7=Zheng |first7=Zhi |last8=Han |first8=Xu |last9=Liu |first9=Zhiyuan}}</ref> considered the parameter efficiency ("density") of models over time. The idea is that over time, researchers would discover models that use their parameters more efficiently, in that models with the same performance can have fewer parameters.

A model can have an actual parameter count <math>N</math>, defined as the actual number of parameters in the model, and an "effective" parameter count <math>\hat N</math>, defined as how many parameters it would have taken a previous well-known model to reach he same performance on some benchmarks, such as [[MMLU]]. <math>\hat N</math> is not measured directly, but rather by measuring the actual model performance <math>S</math>, then plugging it back to a previously fitted scaling law, such as the Chinchilla scaling law, to obtain what <math>\hat N</math> would be required to reach that performance <math>S</math>, according to that previously fitted scaling laws.

A densing law states that <math>\ln \left(\frac{\hat N}{N}\right)_{max} = At + B</math>, where <math>t</math> is real-world time, measured in days.

== See also ==
{{div col|colwidth=18em}}
* [[Large language model]]
* [[Foundation model]]
* [[Artificial general intelligence]]
{{div col end}}

== References ==
<references />

[[Category:Artificial neural networks]]
[[Category:Artificial intelligence]]
[[Category:Deep learning]]
[[Category:Power laws]]
[[Category:Statistical laws]]