The Effect of Bottlenecks on Generalisation in Backpropagation Neural Networks

Abstract

Many modifications have been proposed to improve back-propagation's convergence time and generalisation capabilities. Typical techniques involve pruning of hidden neurons, adding noise to hidden neurons which do not learn, and reducing dataset size. In this paper, we wanted to compare these modifications' performance in many situations, perhaps for which they were not designed. Seven famous UCI datasets were used. These datasets are different in dimension, size and number of outliers. After experiments, we find some modifications have excellent effect of decreasing network's convergence time and improving generalisation capability while some modifications perform much the same as unmodified back-propagation. We also seek to find a combine of modifications which outperforms any single selected modification.