Student Probability Seminar

Analysis of one-hidden-layer neural networks via the resolvent method

Speaker: Vanessa Piccolo, ENS Lyon

Location: Warren Weaver Hall 517

Date: Tuesday, October 11, 2022, 5:40 p.m.

Synopsis:

In this talk, I will introduce the random feature model $M = YY*$ with $Y = f(WX)$ generated by a single-hidden-layer neural network. We will compute its asymptotic spectral density using the resolvent method via a cumulant expansion. In particular, we will see that the Stieltjes transform of the limiting spectral distribution approximately satisfies a quartic self-consistent equation, which is exactly the equation obtained by Pennington-Worah (2018) and Benigni-Péché (2021) with the moment method. Time permitting, I will also discuss the case of additive bias $Y = f(WX + B)$ with $B$ being an independent rank-one Gaussian random matrix, closer modelling the neural network infrastructures encountered in practice. This is joint work with Dominik Schröder (ETH Zürich).