I think the real explanation is much easier: Overfitting. This is a classical problem of neuronal networks where too complex networks tend to be overly specific and thus give worse results.