LOBSTER is a sensitivity-based regularizer,safely usable at training time, designed for mak-ing deep neural network models very sparse.LOBSTER drives some but not all weights to-wards zero: it will shrink those weights forwhich the loss derivative is small, such that manyweights eventually become close to zero. Thosethat are close enough to zero will be deleted, i.e.set to zero.In scenarios such as LeNet-300, LeNet-5, ResNet-32 and ResNet-102 trained on MNIST, Fashion-MNIST, CIFAR-10 and ImageNet, LOBSTERyields a significant and competitive reduction ofthe number of nonzero weights with a minimalcomputation overhead.

LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks

Enzo Tartaglione
;
Andrea Bragagnolo;Marco Grangetto;
2020

Abstract

LOBSTER is a sensitivity-based regularizer,safely usable at training time, designed for mak-ing deep neural network models very sparse.LOBSTER drives some but not all weights to-wards zero: it will shrink those weights forwhich the loss derivative is small, such that manyweights eventually become close to zero. Thosethat are close enough to zero will be deleted, i.e.set to zero.In scenarios such as LeNet-300, LeNet-5, ResNet-32 and ResNet-102 trained on MNIST, Fashion-MNIST, CIFAR-10 and ImageNet, LOBSTERyields a significant and competitive reduction ofthe number of nonzero weights with a minimalcomputation overhead.
Enzo Tartaglione; Andrea Bragagnolo; Marco Grangetto; Skjalg Lepsoy
File in questo prodotto:
File Dimensione Formato  
ICML20.pdf

Accesso aperto

Tipo di file: PREPRINT (PRIMA BOZZA)
Dimensione 312.3 kB
Formato Adobe PDF
312.3 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2318/1737767
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact