Special Session 122: Understanding the Learning of Deep Networks: Expressivity, Optimization, and Generalization

On the Expressivity of Neural Networks and Its Applications

Juncai He
The King Abdullah University of Science and Technology
Saudi Arabia
Co-Author(s):    Jinchao Xu and Lin Li
Abstract:
In this talk, I will present some recent results on the expressivity of neural networks and its applications. First, we will illustrate the connections between linear finite elements and ReLU DNNs, as well as between spectral methods and ReLU$^k$ DNNs. Second, we will share our latest findings regarding the open question of whether DNNs can precisely recover piecewise polynomials of arbitrary order on any simplicial mesh in any dimension. Then, we will discuss a specific result on the optimal expressivity of ReLU DNNs and its application, combining it with the Kolmogorov-Arnold representation theorem. Finally, I will offer a remark on the study of convolutional neural networks from an expressivity perspective.