Machine Learning Theory

A lot is known about learning in the abstract. Statistical learning theory gives bounds on how much data a model needs. Work on training dynamics describes how networks move through their loss landscapes. Scaling laws capture, empirically, how performance grows with size and data. These have developed largely on their own tracks, and they tend to give either loose guarantees or curves fitted after the fact rather than quantities you can predict in advance.

This program works toward a single account that connects them. It starts from the time it takes a system to learn something and treats that time as set by a few measurable properties of the problem: the strength of the signal, the noise corrupting it, the curvature of the loss surface, the number of dimensions involved, and the size of the dataset. Knowing how these combine lets you predict which structures get learned fast, which get learned slowly, and which never get learned at all from the data available. The predictions are tight formulas for real nonlinear learning, close to what training actually does, rather than the linearized models and order-of-magnitude bounds that most theory works with.

The optimal learning rate is the step size that makes the best use of the signal before noise and curvature spoil it. The learning time is how long the whole process takes once you account for the dimension being carried, the signal available, and the curvature along the way. Learnability is the amount of data below which the signal cannot be picked out, so learning never gets started. Cortical development is what the same account looks like when the learner is a brain working under tight limits on data, wiring, and the kinds of updates it can make. The sections that follow take the theory through each of these.

Optimal learning rate

Learning rates are usually found by trial and error and treated as a setting to tune. The theory says there is a right one at each point in training, fixed by the local geometry rather than by search. It is the step that gets the most out of the signal before noise and curvature start to cost more than the step is worth. Take a larger step and the noise in the update dominates; take a smaller one and you leave usable signal unused. The best rate sits where those two pressures balance.

The useful part is that the rate comes apart into three things you can read on their own: the dimension being learned, the signal-to-noise ratio, and the curvature of the landscape. That separation is what makes common practice legible. A single global learning rate is a bet that one number can stand in for all three. Diagonal preconditioning handles the curvature term and leaves the others alone. Changing the batch size moves the noise term. Each familiar trick is doing part of the job of a rate you could, in principle, write down in full.

Learning time

Once the rate at a single point is understood, the more demanding quantity is the total time to train: how many steps it takes to go from the starting point to a learned solution. In this framework that time is written down in advance, as an integral along the path the learner takes.

Reading the expression tells you where training time goes. It accumulates with the effective dimension being carried, grows as the signal weakens relative to the noise, and grows with the curvature the trajectory has to work through. The same quantity covers cases that are usually studied apart: high-dimensional training, the effect of architecture on how hard a problem is to optimize, and the value of a good representation. Each is a question about how much dimension, how little signal, and how much curvature the learner has to pay for along the way.

Learnability

Speed is one question; whether learning happens at all is a sharper one. A learner can only make progress when the data carry enough signal to point the update in the right direction. When they do not, no amount of training helps, because the direction the learner would need to follow is buried in noise.

This sets a threshold on the data. Above it, the learning dynamics track the problem the data are drawn from, and training converges to the right answer. Below it, the signal cannot be told apart from the noise in the available sample, and learning never gets going. It gives sample complexity a concrete local form: the amount of data needed to learn at all, at a given point in the problem.

Cortical development and learning

The same account carries over to brains, which learn under far harder limits than artificial networks: little data, noisy updates, plasticity that is local rather than global, sparse wiring, and development that unfolds in stages. A theory built around signal, noise, curvature, dimension, and data gives a way to ask what those limits are for. Several features of biological learning start to look like sensible responses to the geometry the brain is learning in, rather than separate quirks of wiring.

Bounded connectivity, staged development, critical periods, and the early advantage the brain shows for some kinds of structure all fall into this frame. Each can be read as a way of keeping learning tractable: holding the effective dimension down, protecting the signal, or arranging the order in which things are learned so that the curvature stays manageable. Cortical development, on this view, is the same problem of learning time playing out in a system with strict limits on what it can spend.

Scope of the program

The program covers optimal learning rates, pathwise learning time, finite-data corrections, nonlinear learning dynamics, grouping and effective dimensions, optimiser mismatch, conditioning, high-dimensional representation learning, single-index optimisation on the sphere, and applications to cortical development and brain connectivity. The aim throughout is one theoretical language for learning in nonlinear systems, simple enough to illuminate and broad enough to apply across machine learning and neuroscience.