← All Domains

machine_learning

4 papers · 4 Lean-verified

Architectural Optimizations of the In-Context Gradient-Descent Mechanism
The companion core paper establishes, and machine-checks, a single identity: a transformer's forward pass can implement one gradient-descent step on an implicit least-squares objective (the ICL=GD mechanism).
Verified 3,951 words
Capacity, Scaling, and Grokking from the In-Context Learning = Gradient Descent Mechanism
The companion core paper establishes, and machine-checks, a single identity: a transformer's forward pass can implement one gradient-descent step on an implicit least-squares objective (the ICL=GD mechanism). This satellite asks what that verified identity forces to be true about *representational capacity and scaling*.
Verified 4,655 words
Training Dynamics and Inference Guarantees of the In-Context Gradient-Descent Mechanism
The companion core paper establishes, and machine-checks, a single identity: a transformer's forward pass can implement one gradient-descent step on an implicit least-squares objective (the ICL=GD mechanism).
Verified 3,532 words
When In-Context Learning Implements Gradient Descent: A Learned Mechanism, Mechanically Verified and Empirically Tested
We turn the gradient-descent account of in-context learning (ICL) into machine-checked mathematics and falsifiable predictions about real transformers. The formal target is the linear-attention regression identity: a forward pass can implement one gradient-descent step on an implicit least-squares objective.
Verified 14,988 words