Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods

Mini-batch Stochastic Gradient Descent (SGD) is a standard optimization method for training neural networks and other large-scale parametric models. We study three loss-based mini-batch pruning strategies-Perceptron-Inspired SGD (PISGD), Pruning SGD (PSGD), and No-Release PSGD (NRPSGD)-that reduce the number of per-sample gradient evaluations by deprioritizing low-loss samples during training. The methods are evaluated in Python on two problems: polynomial function approximation and MNIST handwritten-digit classification with a one-hidden-layer neural network. We examine the sensitivity to pruning hyperparameters and compare the three strategies with standard mini-batch SGD in terms of training time and optimization or predictive performance. In the reported MNIST experiments, the pruning methods show competitive or improved time-accuracy and time-loss behavior relative to standard mini-batch SGD, with NRPSGD obtaining the most favorable results in the longer runs. These findings provide proof-of-concept evidence restricted to the experimental settings considered. They indicate that loss-based mini-batch pruning can reduce gradient-computation cost without degrading solution quality in the tested settings. Because the pruning mechanism leaves the parameter-update rule unchanged, it may also be combined with adaptive stochastic optimizers such as RMSProp and Adam.