Special Session 122: Understanding the Learning of Deep Networks: Expressivity, Optimization, and Generalization

Overcoming High-Frequency Challenges: From Shallow to Multi-layer Neural Networks

Shijun Zhang
The Hong Kong Polytechnic University
Hong Kong
Co-Author(s):    Hongkai Zhao, Yimin Zhong, Haomin Zhou
Abstract:
This talk explores the limitations of shallow neural networks in handling high-frequency functions and presents a solution through a novel multi-layer, multi-component neural network (MMNN) architecture. We show how shallow networks act as low-pass filters, struggling with high-frequency components due to machine precision and slow learning dynamics. The MMNN architecture addresses these challenges by efficiently decomposing complex functions, significantly improving accuracy and reducing computational costs. Numerical experiments demonstrate the effectiveness of this approach in capturing fine details in oscillatory functions.