Deep Learning Dictionary - Lightweight Crash Course

Learning Rate Schedules - Deep Learning Dictionary

In neural network training, the learning rate is a fractional value, typically set between 0.01 and 0.0001, for which the gradients are multiplied by before updating the weights with this resulting product. This value is a hyperparameter that we set prior to training.

When we first introduced the concept of a learning rate, we assumed that we set the learning rate and leave it at this static rate for the duration of training.