Investigating the Representing Ability of Piecewise Linear Fully-Connected Neural Networks

Deep Neural Networks are believed to have great representing ability due to the huge amount ofparameters. However, a very deep fully connected(FC) network with linear activations only provides a lineartransformation with a quite small number of effective parameters. Through gradually turning on the strengthof nonlinearity of leaky ReLU activations of a FC network and introducing the picture of continuouspiecewise linear(CPWL) mapping, the representing ability of nonlinear networks can be explained. Somearchitectural components, like bottlenecks, can further restrict the representing ability of a linear network,and this restriction still lives for those nonlinear networks with the same network architecture. We provide aformula estimating the effect bottlenecks of the network restrict its representing ability. In the CPWLpicture, there are domains with linear mappings. We study the boundaries between domains, and explainhow the number of neurons for each layer and the depth of the network effect the boundaries. The dynamicsof the boundaries in the learning phase are studied as well, which might contain signals for upcoming lossdrops in the learning process. Finally, we discuss the relation between the representing ability of thenetwork, the spatial distribution of domains, and the nonlinearity of activations.