Deep Learning Scaling Is Predictable, Empirically

Deep learning scaling is predictable, empirically Hestness et al., arXiv, Dec.2017

With thanks to Nathan Benaich for highlighting this paper in his excellent summary of the AI world in 1Q18

This is a really wonderful study with far-reaching implications that could even impact company strategies in some cases. It starts with a simple question: “how can we improve the state of the art in deep learning?” We have three main lines of attack:

1. We can search for improved model architectures.
2. We can scale computation.
3. We can create larger training data sets.

Read more at The Morning Paper