Gaussian Complexity, Metric Entropy, and the Statistical Learning of Deep Nets

Tuesday, September 17, 2019 - 9:00am - 10:00am
Lind 305
Andrew Barron (Yale University)
For deep nets we examine contraction properties of complexity for each layer of the network. For any ReLU network there is, without loss of generality, a representation in which the sum of the absolute values of the weights into each node is exactly 1, and the input layer variables are multiplied by a value V coinciding with the total variation of the path weights. Implications are given for Gaussian complexity, Rademacher complexity, statistical risk, and metric entropy, all of which are shown to be proportional to V. There is no dependence on the number of nodes per layer, except for the number of original input variables d. For estimation with sub-Gaussian noise the mean square generalization error bounds that can be obtained are of order V [(L + log d)/n]^{1/2}, where L is the number of layers and n is the sample size. Moreover, the potential to drop the dependence on the number of layers is explored.