Finding Trainable Subnetworks in Random Neural Networks

Summary & Methodology Analysis

Neural network pruning reduces parameter counts for efficient inference, but contemporary experience shows that the resulting sparse architectures are difficult to train from the start. This paper investigates whether dense, randomly-initialized neural networks contain sparse subnetworks, called winning tickets, that can be trained in isolation to match the test accuracy of the original network. The methodology involves several steps. First, the authors randomly initialize a neural network using parameters drawn from a distribution. Next, they train the network for a set number of iterations. After training, they prune a percentage of the parameters to create a mask. Finally, they reset the remaining parameters to their original initial values to create the winning ticket. The models and datasets evaluated in this study include MNIST, CIFAR10, Lenet, Conv-2, Conv-4, Conv-6, Resnet-18, and VGG-19. The key result is that winning tickets are less than 10 to 20 percent of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. However, the study has notable limitations. The authors only consider vision-centric classification tasks on smaller datasets like MNIST and CIFAR10, and do not investigate larger datasets like Imagenet due to the high computational intensity of iterative pruning. Furthermore, sparse pruning is the only method evaluated, and the resulting architectures are not optimized for modern libraries or hardware. On deeper networks such as Resnet-18 and VGG-19, iterative pruning cannot find winning tickets unless the networks are trained with learning rate warmup.