I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the user's input.
So, I wonder if there are optimizers that can increase/decrease their overall learning rate and other parameters (such as momentum or even weight decay) autonomously depending on some metric, e.g., validation loss, running average of gradients etc. ?
