LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks

LOBSTER is a sensitivity-based regularizer,safely usable at training time, designed for mak-ing deep neural network models very sparse.LOBSTER drives some but not all weights to-wards zero: it will shrink those weights forwhich the loss derivative is small, such that manyweights eventually become close to zero. Thosethat are close enough to zero will be deleted, i.e.set to zero.In scenarios such as LeNet-300, LeNet-5, ResNet-32 and ResNet-102 trained on MNIST, Fashion-MNIST, CIFAR-10 and ImageNet, LOBSTERyields a significant and competitive reduction ofthe number of nonzero weights with a minimalcomputation overhead.