CINECA IRIS Institutional Research Information System
IRIS è il sistema di gestione integrata dei dati della ricerca (persone, progetti, pubblicazioni, attività) adottato dall'Università degli Studi di Torino.
AperTO è l'archivio istituzionale Open Access destinato a raccogliere, rendere visibile e conservare la produzione scientifica dell'Università degli Studi di Torino.
EnglishRecently, a race towards the simplification of deep networks has begun, showing that it is effectively possible to reduce the size of these models with minimal or no performance loss. However, there is a general lack in understanding why these pruning strategies are effective. In this work, we are going to compare and analyze pruned solutions with two different pruning approaches, one-shot and gradual, showing the higher effectiveness of the latter. In particular, we find that gradual pruning allows access to narrow, well-generalizing minima, which are typically ignored when using one-shot approaches. In this work we also propose PSP-entropy, a measure to understand how a given neuron correlates to some specific learned classes. Interestingly, we observe that the features extracted by iteratively-pruned models are less correlated to specific classes, potentially making these models a better fit in transfer learning approaches.
| Titolo: | Pruning Artificial Neural Networks: A Way to Find Well-Generalizing, High-Entropy Sharp Minima |
| Autori Riconosciuti: | |
| Autori: | Tartaglione E.; Bragagnolo A.; Grangetto M. |
| Data di pubblicazione: | 2020 |
| Abstract: | Recently, a race towards the simplification of deep networks has begun, showing that it is effectively possible to reduce the size of these models with minimal or no performance loss. However, there is a general lack in understanding why these pruning strategies are effective. In this work, we are going to compare and analyze pruned solutions with two different pruning approaches, one-shot and gradual, showing the higher effectiveness of the latter. In particular, we find that gradual pruning allows access to narrow, well-generalizing minima, which are typically ignored when using one-shot approaches. In this work we also propose PSP-entropy, a measure to understand how a given neuron correlates to some specific learned classes. Interestingly, we observe that the features extracted by iteratively-pruned models are less correlated to specific classes, potentially making these models a better fit in transfer learning approaches. |
| Editore: | Springer Science and Business Media Deutschland GmbH |
| Titolo del libro: | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
| Volume: | 12397 |
| Pagina iniziale: | 67 |
| Pagina finale: | 78 |
| Nome del convegno: | 29th International Conference on Artificial Neural Networks, ICANN 2020 |
| Luogo del convegno: | svk |
| Anno del convegno: | 2020 |
| Digital Object Identifier (DOI): | 10.1007/978-3-030-61616-8_6 |
| ISBN: | 978-3-030-61615-1 978-3-030-61616-8 |
| Parole Chiave: | Deep learning; Entropy; Post synaptic potential; Pruning; Sharp minima |
| Appare nelle tipologie: | 04A-Conference paper in volume |
File in questo prodotto:
| File | Descrizione | Tipologia | Licenza | |
|---|---|---|---|---|
| ICANN20.pdf | POSTPRINT (VERSIONE FINALE DELL’AUTORE) | Open Access Visualizza/Apri |
Copyright © 2021