Mathematics Colloquium

---

Online hyperparameter optimization by real-time recurrent learning

Speaker: Kyunghyun Cho, NYU

Location: Online

Videoconference link: https://nyu.zoom.us/rec/share/oELMV5r-8dS0EEDbik_ZIG-wa_YYPQM2R1lfg-9z55TpeqQ7iHEdejQknVtbcKs.reCOl-F0hIHxHL0I

Date: Monday, April 26, 2021, 3:45 p.m.

Synopsis:

Conventional hyperparameter optimization methods are computationally intensive and hard to generalize to scenarios that require dynamically adapting hyperparameters, such as life-long learning. Here, we propose an online hyperparameter optimization algorithm that is asymptotically exact and computationally tractable, both theoretically and practically. Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in recurrent neural networks (RNNs). It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously, without repeatedly rolling out iterative optimization. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time. (this is work done with Daniel Jiwoong Im and Cristina Savin.)

Notes: