LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.

LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks

Tartaglione, Enzo;Bragagnolo, Andrea;Fiandrotti, Attilio;Grangetto, Marco
2022

Abstract

LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.
146
230-237
237
Deep learning; Pruning; Regularization; Sparsity
Tartaglione, Enzo; Bragagnolo, Andrea; Fiandrotti, Attilio; Grangetto, Marco
File in questo prodotto:
File Dimensione Formato  
1-s2.0-S0893608021004706-main.pdf

non disponibili

Tipo di file: PDF EDITORIALE
Dimensione 879.22 kB
Formato Adobe PDF
879.22 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/2318/1825747
Citazioni
  • ???jsp.display-item.citation.pmc??? 0
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
social impact