The Propensity for Density in Feed-forward Models

Abstract

This study investigates the propensity for density in deep neural networks, examining the extent to which over-parameterised models retain redundant connections throughout training. We evaluate several pruning and sparsification strategies across a variety of architectures and benchmark datasets, measuring both the achievable compression ratios and the downstream impact on generalisation performance. Our findings suggest that standard training procedures consistently produce networks that are denser than necessary, and that structured pruning methods can recover significant sparsity without meaningful loss in accuracy — raising questions about the efficiency of conventional training paradigms.