Definition of Grokking:<\/strong> Grokking<\/em> refers to a surprising phenomenon of delayed generalization<\/strong> in neural network training. A model will perfectly fit the training data (near-100% training accuracy)<\/strong> yet remain at chance-level on the test set for an extended period. Then, after many more training iterations, the test performance suddenly jumps to near-perfect<\/strong> (sometimes almost overnight) even though training loss had long convergedar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>. This was first clearly documented by Power et al. (2022) on small algorithmic tasks (e.g. modular arithmetic). They showed that with small synthetic datasets (like learning modular addition or multiplication tables), neural networks can memorize the training set quickly but only \u201cgrok\u201d the underlying pattern much later<\/strong>, exhibiting an abrupt transition from overfitting to generalizationar5iv.labs.arxiv.org<\/a>. Crucially, this jump occurs well past the point of overfitting<\/em> \u2013 e.g. thousands of optimization steps after training accuracy is 100%ar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>. Grokking is thus characterized by extreme training time disparity<\/strong>: training loss reaches near-zero early, but validation loss\/accuracy only improves dramatically after a long plateauar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>.<\/p>\n\n\n\n Initial Discovery (Power et al., 2022):<\/strong> In their seminal work, Power et al. trained small transformers on binary operation tables (e.g. modular addition or division). They observed that for sufficiently small datasets, validation accuracy remained at random chance even after training accuracy was perfect, then spiked to 100% after extended training<\/strong>ar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>. They coined this late generalization \u201cgrokking.\u201d<\/em> They also found that smaller datasets require dramatically more training steps to grok<\/strong> (sometimes a 1% decrease in data required ~50% more steps to generalize)ar5iv.labs.arxiv.org<\/a>. Notably, regularization (especially weight decay)<\/strong> was found to accelerate grokking<\/em>ar5iv.labs.arxiv.org<\/a> \u2013 with weight decay, the time to generalize was reducedar5iv.labs.arxiv.org<\/a>. Power et al.\u2019s results established grokking as a real phenomenon and suggested that these toy tasks are a \u201cfertile ground\u201d for studying generalization beyond memorization<\/strong>ar5iv.labs.arxiv.org<\/a>. The discovery raised fundamental questions: Why can a network eventually generalize perfectly without new data<\/em> or changes in training loss? What evolves internally during the long plateau? These questions spurred numerous follow-up studies.<\/p>\n\n\n\n Key Characteristics:<\/strong> To summarize, the hallmark features of grokking are: (1) Delayed generalization:<\/strong> a long period of overfitting (training performance \u226b test performance) followed by a sudden test performance jumpar5iv.labs.arxiv.org<\/a>. (2) Sharp phase transition:<\/strong> the improvement in validation accuracy is rapid once it begins (resembling a phase change rather than gradual improvement)ar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>. (3) Small data regime:<\/strong> grokking is easiest to observe on relatively small or algorithmic datasets where the network can eventually learn<\/em> the structure instead of just brute-force memorizingar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>. (4) Dependence on hyperparameters:<\/strong> factors like weight decay, learning rate, and model capacity influence whether and when grokking occursar5iv.labs.arxiv.org<\/a>ar5iv.labs.arxiv.org<\/a>. For example, high weight decay tends to encourage the model to eventually find a simpler, generalizing solution rather than sticking to a memorizing solutionar5iv.labs.arxiv.org<\/a>. Grokking was initially documented in small transformers on tasks like modular arithmeticar5iv.labs.arxiv.org<\/a>, but subsequent work (discussed below) has found analogous behavior in other architectures and even non-neural models, indicating it taps into a general phenomenon about learning dynamics.<\/p>\n\n\n\n Researchers have proposed several theoretical frameworks to explain why<\/em> grokking happens. These include views based on training dynamics regimes<\/strong>, representation learning \u201cphases,\u201d<\/strong> mechanistic circuit formation,<\/strong> and statistical distribution shifts<\/strong>. We outline the major explanations:<\/p>\n\n\n\n (a) Lazy-to-Rich Training Dynamics (Kumar et al., 2024):<\/strong> Kumar and colleagues interpret grokking as a consequence of a neural network transitioning from a \u201clazy\u201d learning regime to a \u201crich\u201d feature-learning regimeopenreview.net<\/a>openreview.net<\/a>. In the lazy regime<\/em>, the network\u2019s parameters change only minimally (akin to a kernel or linear model), so the network initially fits the training data using its fixed initial features<\/em> (essentially performing something like kernel regression)openreview.net<\/a>openreview.net<\/a>. This can drive training loss down (memorizing the training set) but doesn\u2019t generalize if the initial random features aren\u2019t aligned with the true patternopenreview.net<\/a>openreview.net<\/a>. After this, a late-phase transition<\/em> happens: the network eventually leaves the lazy regime and begins genuine feature learning<\/strong> (the \u201crich\u201d regime)openreview.net<\/a>openreview.net<\/a>. At this point it discovers new representations that capture the underlying structure, leading to a sudden generalization jump. Kumar et al. demonstrated this mechanism in a simple two-layer network on a polynomial task that groks even without<\/em> regularizationarxiv.org<\/a>arxiv.org<\/a>. They identified conditions for grokking: (1) Misaligned initial kernel:<\/strong> the top eigenfunctions of the Neural Tangent Kernel are not aligned with the target function, so initial learning struggles to generalizeopenreview.net<\/a>openreview.net<\/a>. (2) Intermediate dataset size:<\/strong> enough data that generalization is eventually possible, but not so much that test performance tracks training performance from the startopenreview.net<\/a>openreview.net<\/a>. (3) Small initial step size (or network scaling):<\/strong> ensuring training starts in the lazy\/linear regime rather than immediately learning featuresopenreview.net<\/a>. Under these conditions, the network first fits training data like a linear model (lazy phase), then later switches to learning new features<\/em> that solve the task (rich phase), which coincides with grokkingopenreview.net<\/a>openreview.net<\/a>. This theory connects grokking to classical ideas of neural tangent kernels vs. feature learning<\/strong>. Indeed, they provided evidence in settings like MNIST and simple transformers that delaying feature learning can induce grokkingopenreview.net<\/a>openreview.net<\/a>. In short, grokking = the moment the network stops being a mere \u201clazy\u201d interpolator and actually learns the true features<\/strong>, causing test accuracy to leapopenreview.net<\/a>openreview.net<\/a>.<\/p>\n\n\n\n (b) Effective Theory and the \u201cGoldilocks Zone\u201d (Liu et al., 2022):<\/strong> Another perspective comes from Liu et al., who developed an effective theory of representation learning<\/strong> to map out different \u201cphases\u201d of trainingpapers.neurips.cc<\/a>papers.neurips.cc<\/a>. They ran extensive experiments on algorithmic tasks and identified four regimes: confusion<\/strong>, memorization<\/strong>, grokking<\/strong>, and comprehension<\/strong>papers.neurips.cc<\/a>. \u2013 Confusion<\/em> means the model fails to even memorize (training loss high). Memorization<\/em> means it can fit training data but not generalize. Comprehension<\/em> means it generalizes quickly (no long delay). Grokking<\/em> is the in-between case of delayed generalizationpapers.neurips.cc<\/a>papers.neurips.cc<\/a>. Crucially, they found representation learning only happens in a \u201cGoldilocks zone\u201d between pure memorization and pure confusion<\/strong>papers.neurips.cc<\/a>papers.neurips.cc<\/a>. In this zone (which includes the comprehension and grokking phases), the model learns structured internal representations that enable generalizationpapers.neurips.cc<\/a>papers.neurips.cc<\/a>. If hyperparameters are tuned for too-rapid learning or too slow, the model either trivially memorizes or never learns anything useful (confusion). But just-right settings (not too much data or too high learning rate) force the model to \u201cstruggle\u201d and in doing so, discover structurepapers.neurips.cc<\/a>papers.neurips.cc<\/a>. They observed that in transformers, the grokking phase sits closer to memorization<\/em> on this spectrum, hence generalization is delayedpapers.neurips.cc<\/a>. Intuitively, grokking is an \u201cundesirable\u201d phase caused by slightly improper hyperparameters that slow representation learningpapers.neurips.cc<\/a>. By charting phase diagrams of training dynamics (varying e.g. learning rate for embeddings vs. decoder, or weight decay), they showed one can see regions corresponding to each phasepapers.neurips.cc<\/a>papers.neurips.cc<\/a>. Notably, grokking appears sandwiched between the fast-generalizing comprehension phase and the pure memorization phase<\/strong>papers.neurips.cc<\/a>. If one tunes hyperparameters toward comprehension (e.g. a somewhat faster or more flexible representation learning), the model can \u201cde-delay\u201d generalization and avoid the grokking plateaupapers.neurips.cc<\/a>. This framing suggests grokking is not a mysterious anomaly but a point on a continuum of learning behaviors, explainable by the competition between learning representations vs. overfitting weights<\/em>. They even likened the effect to \u201cintelligence from starvation\u201d<\/em> in evolution \u2013 i.e., resource limitations (small data, weight decay) force the network to find a more efficient solution<\/strong> once pure memorization failspapers.neurips.cc<\/a>. Overall, Liu et al. provided an intuitive phase diagram: grokking happens when a network eventually exits a near-memorization phase to find structure, and careful tuning can eliminate the delaypapers.neurips.cc<\/a>papers.neurips.cc<\/a>.<\/p>\n\n\n\n (c) Phase Transitions & Mechanistic Interpretability (Nanda et al., 2023):<\/strong> Some researchers approached grokking by reverse-engineering the network\u2019s internal mechanism<\/em> on a grokking task. Nanda et al. performed a detailed mechanistic interpretability analysis of a small transformer trained on modular addition<\/strong>, to see what changes inside the model during grokking<\/em>arxiv.org<\/a>. They found that the model actually learns a sensible algorithm: it represents numbers in a Fourier basis and performs addition via rotations on the circle (a known algorithm for modular addition)arxiv.org<\/a>. Importantly, they identified three continuous \u201cphases\u201d in training: (1) Memorization phase:<\/strong> the network first stores some training mappings (e.g. lookup table behavior) to drive training loss down. (2) Circuit formation phase:<\/strong> the network gradually builds up the correct algorithmic circuit<\/strong> (sine\/cosine representations for numbers, etc.) while still mostly memorizingarxiv.org<\/a>. (3) Cleanup phase:<\/strong> the network removes or downweights the memorization-based components<\/em> and relies on the general algorithm, which suddenly boosts test accuracyarxiv.org<\/a>. In this view, grokking is not actually a \u201cmagic\u201d sudden insight but the result of a gradual improvement in the learned representations\/circuits that isn\u2019t reflected in test performance until a tipping point<\/strong>arxiv.org<\/a>. Their progress measures showed that the \u201cgeneral\u201d circuit\u2019s strength grows continuously and the \u201cmemorization\u201d circuits decay, and the test accuracy jump happens when the general circuit\u2019s signal dominatesarxiv.org<\/a>. Thus, what looks like a phase transition from the outside is underpinned by continuous changes internally<\/em>. They conclude that grokking = the emergent result of two competing solution circuits<\/strong> \u2013 one brute-force memorizing, one generalizing \u2013 where the generalizing circuit eventually wins outarxiv.org<\/a>. This aligns with earlier informal conjectures (e.g. by Millidge 2022, Shah 2021) that grokking may involve the network first finding a shortcut (memorization), then later discovering the true pattern<\/em> and switching overpapers.neurips.cc<\/a>. Mechanistic studies by Nanda et al. and others also highlight the role of weight decay<\/strong>: weight decay preferentially suppresses complex memorizing solutions, effectively encouraging the network to find the more structured algorithm soonerar5iv.labs.arxiv.org<\/a>. In summary, a mechanistic view treats grokking as a competitive phase transition<\/strong> between different \u201ccircuits\u201d in the model \u2013 when the simplistic memorization falls away and the elegant general solution crystallizes, we see a sudden performance jumparxiv.org<\/a>.<\/p>\n\n\n\n (d) Statistical and Distribution-Shift Explanations (Carvalho et al., 2025):<\/strong> A recent line of work frames grokking as a statistical phenomenon related to distribution shift<\/em>. Carvalho et al. argue that a key factor behind grokking is a mismatch between the training distribution and the test distribution<\/strong>arxiv.org<\/a>arxiv.org<\/a>. In their view, small or biased training sets create implicit distribution shifts<\/em> that the model must eventually recognize. They formalize \u201clate generalization\u201d and demonstrate grokking on carefully constructed synthetic datasets that manipulate class\/subclass distributionsarxiv.org<\/a>arxiv.org<\/a>. For example, they create a dataset where each class has two subclasses. If one subclass is sparsely sampled in training, the model initially overfits (focusing on the dominant subclass), but much later it learns to leverage relationships between subclasses, allowing generalization to the sparse subclass<\/strong>arxiv.org<\/a>arxiv.org<\/a>. This manifests as grokking. By controlling sampling imbalance, they could induce or prevent grokking at will (e.g. removing<\/em> a subclass entirely prevented late generalization, while weakly sampling<\/em> it caused grokking)arxiv.org<\/a>arxiv.org<\/a>. They conclude that data sparsity alone isn\u2019t the direct cause<\/strong> \u2013 rather, data sparsity causes an implicit shift that the model only overcomes after learning higher-level relations<\/em>arxiv.org<\/a>arxiv.org<\/a>. Interestingly, they show grokking can occur even with dense data and minimal hyper-parameter tuning<\/em>, contrary to early beliefs that you needed extremely tiny data and heavy regularizationarxiv.org<\/a>. They also extended experiments to a real-world scenario (inducing distribution shift on MNIST digits via clustering distortions) and observed delayed generalization there as wellarxiv.org<\/a>arxiv.org<\/a>. This statistical view connects grokking with phenomena like domain adaptation<\/em>: the training set is not fully representative, and only after memorizing does the model find a more general decision boundary that works for the broader distributionarxiv.org<\/a>arxiv.org<\/a>. One practical outcome they stress is the need for better early-stopping or progress metrics<\/strong> \u2013 since traditional validation loss might be flat during the grokking plateau, one might prematurely stop trainingarxiv.org<\/a>arxiv.org<\/a>. Their work \u201cpaves the way for developing better stopping criteria\u201d by understanding how distribution structure cues late generalizationarxiv.org<\/a>. In essence, the Carvalho et al. explanation is: grokking happens when the training data is just sufficient to eventually infer a general rule, but not initially sufficient to show that generalization on the test distribution is possible<\/strong> \u2013 the model initially fits idiosyncrasies, then slowly figures out the underlying pattern connecting train and test distributionsarxiv.org<\/a>arxiv.org<\/a>.<\/p>\n\n\n\n Each of these frameworks sheds light on grokking from different angles (dynamics, representations, circuits, data distribution). They aren\u2019t mutually exclusive; indeed, there\u2019s a sense in which \u201ccircuits competition\u201d<\/strong>, \u201clazy-to-rich transition\u201d<\/strong>, and \u201cdistribution shift recognition\u201d<\/strong> might be describing the same core process at different levels. The next framework attempts to unify some of these ideas.<\/p>\n\n\n\n (e) Unified \u201cCircuits Competition\u201d Framework (Huang et al., 2024):<\/strong> Huang et al. propose a unifying view connecting grokking with double descent<\/strong> (the puzzling re-improvement of test error when model size grows) and emergent abilities in LLMs<\/strong>arxiv.org<\/a>arxiv.org<\/a>. They build on the idea of memorization vs. generalization circuits competing inside the model<\/strong>arxiv.org<\/a>arxiv.org<\/a>. This perspective was initially used for grokking (as discussed by Nanda et al.), but Huang et al. extend it across different scales of model size and data sizearxiv.org<\/a>arxiv.org<\/a>. Their framework outlines four training dynamics regimes<\/em> depending on model capacity and data: for instance, (1) small model + little data = can\u2019t even memorize (underfitting), (2) large model + little data = memorization (overfitting), (3) intermediate model\/data = grokking (late generalization), (4) large model + ample data = immediate generalization (comprehension)arxiv.org<\/a>arxiv.org<\/a>. This mirrors the phases discussed by Liu et al., but explicitly ties them to model size and the classic double-descent curve. Using this framework, they reinterpret double descent: as model size increases, initially a model memorizes (test error high), but beyond a critical size it can implement generalizing circuits, causing test error to drop again (second descent)arxiv.org<\/a>arxiv.org<\/a>. They made two predictions about when double descent occurs<\/strong> based on this circuits competition, and verified them experimentallyarxiv.org<\/a>arxiv.org<\/a> (for example, predicting the model size at which memorization gives way to generalization given a fixed data size). Furthermore, they extended the idea to multi-task learning and emergent abilities<\/strong>. By treating an \u201calgorithmic\u201d task as an emergent ability<\/em> (e.g. a task that only larger models learn, analogous to tasks that only \u201cemerge\u201d in very large LLMs), they show that the same framework predicts when a task will suddenly be learned (emerge) as data or model scale increasesarxiv.org<\/a>. In other words, an emergent ability<\/em> can be seen as grokking in the context of many tasks: smaller models effectively memorize training tasks without discovering the algorithm for the \u201cemergent\u201d task, until a certain scale where the circuit for that task can form and win outarxiv.org<\/a>. This offers a novel theoretical lens on why large language models suddenly acquire new skills at scale \u2013 it\u2019s the outcome of internal competition between specialized circuits, analogous to grokking dynamics. Overall, Huang et al.\u2019s contribution is a conceptual synthesis<\/strong>: grokking, double descent, and emergent capabilities are all manifestations of the same underlying dynamic<\/em> of memorization vs. generalization circuits, playing out over different axes (time, model size, or task complexity)arxiv.org<\/a>arxiv.org<\/a>. This unified view helps tie the grokking literature into the broader context of generalization phenomena in deep learning.<\/p>\n\n\n\n Initially, grokking was studied in small transformers on synthetic tasks. A natural question is: does grokking occur in other models, or is it unique to deep neural networks (and to those specific tasks)? Recent research indicates grokking is more general<\/strong> \u2013 it appears in various model families and deeper networks, though sometimes in modified forms.<\/p>\n\n\n\n Grokking in Non-Neural Models (Miller et al., 2023):<\/strong> Strikingly, Miller et al. showed that grokking is not<\/em> exclusive to neural nets<\/strong> \u2013 they observed analogous delayed generalization in models like Gaussian Processes (GPs)<\/strong>, simple linear regression<\/strong>, and Bayesian neural networks<\/strong>arxiv.org<\/a>arxiv.org<\/a>. In their study, even a Gaussian process classifier<\/em> trained on a small algorithmic dataset displayed a grokking-like jump in test performance after more data was effectively consideredarxiv.org<\/a>. Likewise, linear regression on a certain structured task showed a form of late generalizationarxiv.org<\/a>. This is surprising because these models don\u2019t undergo iterative \u201cfeature learning\u201d in the neural sense. Miller et al. argue the common factor is that these learning methods can be implicitly guided by a notion of solution complexity vs. error trade-off<\/strong>arxiv.org<\/a>. For example, in GP regression with a certain kernel, the function that fits the training data initially might be very wiggly (memorizing), but as the GP posterior updates (or as hyperparameters favor smoother functions), a simpler generalizing function can eventually dominate \u2013 analogous to a late shift to a low-complexity solutionarxiv.org<\/a>. They even devised a way to induce grokking behavior<\/em> in algorithmic tasks by adding spurious input dimensions: extra random features that the model can memorize initially, forcing a delay until it learns to ignore them and focus on real featuresarxiv.org<\/a>. The key takeaway is any learning system that balances model complexity and fit could exhibit grokking<\/strong>arxiv.org<\/a>. Grokking is \u201cnot restricted to settings considered in current theoretical and empirical studies\u201d \u2013 it may arise \u201cin any model where solution search is guided by complexity and error\u201d<\/em>arxiv.org<\/a>. In simpler terms, if a learning algorithm can first find a more complex solution that fits the data and only later shift to a simpler, generalizing solution, it can grok. This finding broadens the scope: delayed generalization is not<\/em> a quirk of backpropagation or transformers, but a potential phenomenon in general learning dynamics, including Bayesian paradigms. It invites theoretical analysis of, say, double descent in kernel methods and how that might relate to grokking. (Indeed, double descent in linear models could be seen as a cousin of grokking, where varying model complexity yields non-monotonic generalization.)<\/p>\n\n\n\n Deep Neural Networks and Multi-Stage Generalization (Fan et al., 2024):<\/strong> Earlier grokking studies mostly used shallow networks (e.g. a 1-layer transformer). Fan et al. asked: what happens in deeper<\/em> neural networks? They trained deeper MLPs (up to 12 layers) on algorithmic tasks and found that deeper networks not only grok, but can exhibit multiple grokking stages<\/em><\/strong>arxiv.org<\/a>arxiv.org<\/a>. Specifically, a 12-layer network sometimes showed two<\/em> distinct jumps in test accuracy: an initial delayed jump, then after further training, a second surge in performancearxiv.org<\/a>. This \u201cmulti-stage generalization\u201d<\/strong> was rarely seen in shallow modelsarxiv.org<\/a>. It suggests that deeper models might learn complex tasks in a hierarchical fashion \u2013 e.g. first grok some simpler aspect, then later grok a finer aspect. Correspondingly, Fan et al. measured the internal feature rank<\/strong> (roughly, the dimensionality of the learned representations) over trainingarxiv.org<\/a>. They observed that as grokking occurs, the feature rank of intermediate layers drops<\/em><\/strong>, indicating the network\u2019s representations become more compressed and structuredarxiv.org<\/a>. Intriguingly, the feature rank trajectory often showed a double-descent shape<\/strong>: it would decrease, then increase, then decrease again, aligning with the two surges in accuracyarxiv.org<\/a>. In other words, when test accuracy made a second leap, it coincided with another drop in feature rank complexity. These findings hint that internal representation compression is a signature of generalization<\/strong> \u2013 the network throws away redundant\/memorized information and distills a more low-dimensional concept, which yields better generalizationarxiv.org<\/a>. They even suggest that feature rank might predict grokking better than weight norm or other metrics<\/em>arxiv.org<\/a>. Practically, one could monitor feature rank during training as an unsupervised indicator of an impending generalization jumparxiv.org<\/a>. Fan et al.\u2019s work expands grokking research to deeper architectures<\/em>, showing that depth can increase the propensity to grok (they found deep nets were more susceptible<\/em> to grokking than shallow ones)arxiv.org<\/a>. It also reinforces connections between grokking and double descent<\/strong>: the double descent in feature complexity mirrors a kind of double descent in performancearxiv.org<\/a>. In summary, deep networks can grok in potentially multiple phases, and studying their layerwise representations (like rank) can reveal how generalization emerges internally across training stagesarxiv.org<\/a>.<\/p>\n\n\n\n Unified Perspectives: Grokking, Double Descent, Emergence (Huang et al., 2024):<\/strong> As mentioned earlier, Huang et al. provide a framework that unifies these phenomena by focusing on circuits competition across scalesarxiv.org<\/a>arxiv.org<\/a>. In the context of different model types: double descent<\/em> has been observed in linear models and random forests, and emergent abilities<\/em> are discussed for LLMs \u2013 by showing these can all be seen through the lens of late-forming generalist circuits overtaking memorizing ones, they argue grokking is a widespread dynamic. Huang et al. delineated how increasing model capacity or data can move a model between the four regimes (no-fit, memorize, grok, comprehend)arxiv.org<\/a>. This suggests, for example, that a sufficiently large model might not grok (it jumps straight to generalization, the comprehension phase) \u2013 which might explain why grokking is harder to notice in very large-scale settings unless carefully analyzed. But in intermediate regimes (including many practical scenarios), we might expect some grokking-like behaviors.<\/p>\n\n\n\n Other Model-Agnostic Findings:<\/strong> Additional works have explored simplified theoretical models of grokking. For instance, Levi et al. (2023)<\/em> analyzed a linear estimator that groks<\/strong> \u2013 they constructed a solvable setup (linear regression with particular features) where the solution exhibits a delayed generalization effect, giving a fully analytical handle on grokking dynamics. Lyu et al. (2023)<\/em> proved in a theoretical setting that a \u201cdichotomy of early vs. late implicit bias\u201d in gradient descent could provably<\/em> lead to grokking-like behavior (early training minimizes training error in one way, later dynamics of gradient descent shift the solution towards a different minimum with better generalization)gwern.net<\/a>. These works support the notion that grokking can emerge from general properties of optimization in high-dimensional systems, not just quirky neural network tricks.<\/p>\n\n\n\n The upshot is that grokking generalizes beyond its initial context<\/strong>. It has been replicated in kernel methods, probabilistic models, and deeper nets, and connected to phenomena like double descent and phase transitions. This broadens confidence that grokking reveals something fundamental about learning: there can be multiple qualitatively different regimes during training<\/em>, and the final generalization behavior may be decided by a late-time dynamical shift<\/strong> rather than evident early on.<\/p>\n\n\n\n A critical question is whether grokking occurs in large-scale, real-world training<\/em>, such as the pretraining of large language models (LLMs). For a long time, grokking was mostly a toy-setting curiosity. Recent studies in 2025 provide evidence that grokking does manifest during LLM pretraining \u2013 albeit in a more complex, asynchronous way \u2013 and that we can detect it via the model\u2019s internal dynamics<\/strong>.<\/p>\n\n\n\n Grokking in Mixture-of-Experts LLM (Li et al., 2025):<\/strong> Li and colleagues conducted the first study of grokking in the context of a full-scale LLM pretraining run<\/em>. They analyzed checkpoints from the training of OLMoE<\/strong>, a 7-billion-parameter Mixture-of-Experts transformer (so, a large model with multiple experts per layer)arxiv.org<\/a>arxiv.org<\/a>. Crucially, they did not have a traditional held-out test set to watch accuracy during pretraining<\/strong> (since language model pretraining is unsupervised), so they crafted a methodology: they computed the model\u2019s loss on its training data over time and also periodically evaluated the emergence of capabilities<\/em> on various downstream tasks (math reasoning, code generation, factual QA) using intermediate checkpointsarxiv.org<\/a>arxiv.org<\/a>. They indeed found that grokking-like delayed generalization happens in LLM pretraining<\/strong>arxiv.org<\/a>. However, unlike the toy tasks where suddenly all<\/em> data is grokked at once, in LLMs different domains or skill areas grokked at different times<\/strong>arxiv.org<\/a>arxiv.org<\/a>. For example, perhaps the model\u2019s performance on math word problems might remain low until very late in training (a \u201clater grokking\u201d capability), whereas its performance on commonsense QA might improve earlier. They called this \u201clocal grokking\u201d<\/em> \u2013 each subset of the training data (or each domain\/task) has its own delayed generalization pointarxiv.org<\/a>. Early in pretraining, the model\u2019s generalization (when evaluated on downstream tasks) was unstable<\/em>, improving on some tasks then dropping, etc., which they attribute to these asynchronous grokking events across domainsarxiv.org<\/a>. Once sufficient data had been seen and memorized in a domain, that domain\u2019s test performance started improving steadily<\/em>arxiv.org<\/a>. Notably, more difficult data (or tasks) grokked later and had longer delays<\/strong>, which aligns with intuition \u2013 complex patterns take longer for the model to discover even after fitting the easier partsarxiv.org<\/a>arxiv.org<\/a>.<\/p>\n\n\n\n Routing Dynamics as Generalization Indicators:<\/strong> Because evaluating an LLM on test tasks in the middle of pretraining is expensive and confounded (since the model isn\u2019t instruction-tuned yet), Li et al. proposed to monitor internal model metrics<\/strong> instead. In a Mixture-of-Experts (MoE) model, a routing network<\/em> directs each input to certain expert sub-networks at each layer. Li et al. tracked the expert choice patterns (pathways) for training samples<\/strong> throughout trainingarxiv.org<\/a>arxiv.org<\/a>. They discovered an intriguing mechanistic change: during grokking, Theoretical Explanations and Mechanistic Models<\/h1>\n\n\n\n
Applicability and Generalization Across Model Types<\/h1>\n\n\n\n
Applications in LLM Pretraining<\/h1>\n\n\n\n