Solving a classification problem for a neural network means looking for a particular configuration of the internal parameters. This is commonly achieved by minimizing non-convex object functions. Hence, the same classification problem is likely to have several, different, equally valid solutions, depending on a number of factors like the initialization and the adopted optimizer. In this work, we propose an algorithm which looks for a zero-error path joining two solutions to the same classification problem. We witness that finding such a path is typically not a trivial problem; however, our heuristics is able to succeed in such a task. This is a step forward to explain why simple training heuristics (like SGD) are able to train complex neural networks: we speculate they focus on particular solutions, which belong to a connected solution sub-space. We work in two different scenarios: a synthetic, unbiased and totally-uncorrelated (hard) training problem, and MNIST. We empirically show that the algorithmically-accessible solutions space is connected, and we have hints suggesting it is a convex sub-space. © 2019, Springer Nature Switzerland AG.
Titolo: | Take a Ramble into Solution Spaces for Classification Problems in Neural Networks |
Autori Riconosciuti: | |
Autori: | Tartaglione, Enzo; Grangetto, Marco |
Data di pubblicazione: | 2019 |
Abstract: | Solving a classification problem for a neural network means looking for a particular configuration of the internal parameters. This is commonly achieved by minimizing non-convex object functions. Hence, the same classification problem is likely to have several, different, equally valid solutions, depending on a number of factors like the initialization and the adopted optimizer. In this work, we propose an algorithm which looks for a zero-error path joining two solutions to the same classification problem. We witness that finding such a path is typically not a trivial problem; however, our heuristics is able to succeed in such a task. This is a step forward to explain why simple training heuristics (like SGD) are able to train complex neural networks: we speculate they focus on particular solutions, which belong to a connected solution sub-space. We work in two different scenarios: a synthetic, unbiased and totally-uncorrelated (hard) training problem, and MNIST. We empirically show that the algorithmically-accessible solutions space is connected, and we have hints suggesting it is a convex sub-space. © 2019, Springer Nature Switzerland AG. |
Editore: | Springer Verlag |
Titolo del libro: | International Conference on Image Analysis and Processing, ICIAP 2019 |
Volume: | 11751 |
Pagina iniziale: | 345 |
Pagina finale: | 355 |
Nome del convegno: | International Conference on Image Analysis and Processing, ICIAP 2019 |
Luogo del convegno: | Trento |
Anno del convegno: | 9/9/2019 |
Digital Object Identifier (DOI): | 10.1007/978-3-030-30642-7_31 |
ISBN: | 978-3-030-30641-0 978-3-030-30642-7 |
Appare nelle tipologie: | 04A-Conference paper in volume |
File in questo prodotto:
File | Descrizione | Tipologia | Licenza | |
---|---|---|---|---|
ICIAP19_takearamble.pdf | POSTPRINT (VERSIONE FINALE DELL’AUTORE) | Open Access Visualizza/Apri |