(2018) characterized this phenomenon by a “double descent” curve which extends the classical U-shaped curve. These models are usually trained to achieve near-zero error on the training data, and yet they still have remarkable performance on test data.

However, it is common nowadays to use highly complex over-parameterized models like deep neural networks. This is described by the classical U-shaped curve for the test error when the model complexity varies (see Figure 1(a)). A smaller model would give a larger training error while making the model bigger would result in a larger generalization gap, both leading to larger test errors. There is a “sweet spot” for the model complexity such that the model (i) is big enough to achieve reasonably good training error, and (ii) is small enough so that the generalization gap – the difference between test error and training error – can be controlled. Traditional wisdom in machine learning holds that there is a careful trade-off between training error and generalization gap. (2018) of the effect of increased model complexity on generalization: traditional belief (a) vs actual practice (b).

Ultra model nn set