Correlated activity pruning (CAPing)

The generalisation ability of an Artificial Neural Network (ANN) is dependent on its architecture. An ANN with the correct architecture will learn the task presented by the training set but also acquire rales that are general enough to correctly predict outputs for unseen test set examples. To obtain this optimum network architecture it is often necessary to apply a labourious ‘trial and error’ approach. One approach that helps to achieve optimum network architecture in a more intelligent way is pruning. Such methods benefit from the learning advantages of larger networks while reducing the amount of overtraining or memorisation within these networks. Sietsma and Dow (1988) describe an interactive pruning method that uses several heuristics to identify units that fail to contribute to the solution and therefore can be removed with no degradation in performance. This approach removes units with constant outputs over all the training patterns as these are not participating in the solution. Also, units with identical or opposite activations for all patterns can be combined. The approach to merging hidden units detailed in Sietsma and Dow’s paper is useful, however, it only covers perfectly correlated, binary activations.