There Will Be a Scientific Theory of Deep Learning

The paper "There Will Be a Scientific Theory of Deep Learning" (arXiv:2604.21691) argues that a fundamental scientific theory of deep learning—termed **"learning mechanics"**—is emerging. This theory aims to be the physics of learning: a mathematical framework that predicts coarse, aggregate statistics of training dynamics, representations, and model performance from first principles.

The authors present five lines of evidence supporting this claim, each drawing analogies to physics (mechanics, statistical physics, and thermodynamics):

1. Analytically Solvable Settings Exist

Just as physics uses the harmonic oscillator or hydrogen atom as solvable cornerstones, deep learning admits tractable models that provide intuition for broader phenomena.

• Linearization in data: Deep linear networks exhibit exactly solvable gradient flow dynamics (decoupled Bernoulli ODEs) showing "greedy low-rank learning"—larger singular modes are learned first.
• Linearization in parameters: The Neural Tangent Kernel (NTK) limit describes wide networks as kernel ridge regression, enabling exact predictions of test performance from data statistics.
• Beyond linearization: Recent progress solves nonlinear settings like multi-index models, quadratic activations, and shallow networks in high dimensions.

Figure 1: (Left) Exact solutions for deep linear networks show sequential learning of singular modes. (Right) NTK-based predictions match test performance of wide neural networks.

2. Insightful Limits Reveal Fundamental Behavior

Complex systems simplify in infinite-size limits. Neural networks exhibit distinct "phases" depending on scaling:

• The Lazy/Rich Dichotomy:
  • Lazy regime: With standard initialization ($\sim \text{width}^{-1/2}$), networks behave like frozen kernels (NTK regime).
  • Rich regime: With smaller initialization ($\sim \text{width}^{-1}$), networks exhibit feature learning—hidden representations adapt to data structure.
• Infinite Depth: Residual networks behave like Neural ODEs or stochastic differential equations depending on depth-scaling.
• The Discretization Hypothesis: Finite networks may be understood as discretized approximations to infinite continuous systems (analogous to PDE discretization).

Figure 2: Same network architecture exhibits lazy (weights frozen) vs. rich (weights cluster toward teacher features) dynamics depending on initialization scale.

3. Simple Empirical Laws Capture Macroscopic Statistics

Aggregate observables follow predictable mathematical laws:

• Neural Scaling Laws: Test loss follows power laws in compute, dataset size, and parameter count ($L \propto C^{-\alpha}$). These are predictable but their exponents remain theoretically unexplained.
• Edge of Stability: With gradient descent, the Hessian sharpness (largest eigenvalue) progressively increases then stabilizes near $2/\eta$ (the stability threshold), explaining why training oscillates yet converges.
• Neural Collapse: At convergence, class representations form a regular simplex (tightly clustered class means with maximal angular separation).
• Conservation Laws: Gradient flow exhibits Noetherian conservation laws (e.g., layerwise covariance differences are conserved).

Figure 3 (Top): Power-law scaling of loss with compute/data/parameters. Figure 4 (Bottom): Sharpness stabilizes at the edge of stability ($2/\eta$) across architectures.

4. Hyperparameters Can Be Disentangled and Understood

Optimization and architecture hyperparameters can be mathematically decoupled:

• μP (Maximal Update Parameterization): By scaling learning rates and initializations appropriately with width ($\eta \sim \text{width}^{-1}$), training dynamics remain stable across model sizes, enabling zero-shot hyperparameter transfer from small to large models.
• Implicit Regularization: SGD implicitly penalizes loss curvature (Hessian sharpness), explaining why large learning rates and small batch sizes often generalize better.

Figure 5: Under μP, optimal learning rates remain constant across widths, enabling transfer from small proxy models to large production models.

5. Universal Phenomena Appear Across Settings and Tasks

Similar to critical phenomena in physics, deep learning exhibits universality:

• Universal Representations: Different architectures (CNNs, Transformers) and modalities (vision, language) converge to similar internal representations as scale increases ("Platonic Representation Hypothesis").
• Universal Inductive Biases: Architectures with different designs (e.g., U-Net vs. Transformer diffusion models) produce nearly identical outputs given the same random seeds, suggesting shared fundamental biases.
• Universal Data Structure: Natural data (images, text) shares power-law spectra and hierarchical structure, explaining why the same algorithms work across domains.

Figure 6: (Left) Different diffusion architectures produce identical images from the same noise seeds. (Right) Representation similarity between language and vision models increases with scale.

Synthesis: Learning Mechanics ↔ Mechanistic Interpretability

The authors propose a symbiotic relationship between learning mechanics (the "physics" of aggregate dynamics) and mechanistic interpretability (the "biology" of circuits and features):

• Mechanics → Interpretability: Provides rigorous foundations for assumptions (linearity, locality, sparsity) and explains how mechanisms form during training.
• Interpretability → Mechanics: Empirical discoveries (induction heads, grokking, Fourier features) provide concrete phenomena for theory to explain.

Open Directions

The paper identifies 10 critical open problems, including:

1. Solvable models of genuinely deep, nonlinear learning
2. Theoretical capture of natural data structure
3. Predicting scaling law exponents a priori
4. Formal definitions of "features" and "circuits"
5. Understanding whether finite networks are approximations to infinite limits

The authors conclude that deep learning theory has transitioned from pure mathematics to an empirical science, with the transparency and measurability of neural networks making a comprehensive "learning mechanics" achievable.