Let's say that I want to create an optimization algorithm, which is supposed to find an optimum value for a given objective function. Creating an optimization algorithm to explore through the search space can be quite challenging.
My question is: can machine learning be used to automatically create optimization algorithms? Is there any source to look at for this?
Machine learning has been used to automatically learn new optimization/learning algorithms. This task is often known as meta-learning, i.e. you learn to learn, in this case, an optimization algorithm, but note that meta-learning does not just refer to learning optimization algorithms (see this blog post).
The blog post Learning to Optimize with Reinforcement Learning (2017) is a good introduction to the topic and focuses on the approach proposed in this paper Learning to Optimize (2016), which uses reinforcement learning to solve this problem: more specifically, they learn a policy (in practice, represented as a neural network) that represents the learned optimization algorithm.
There are other related approaches: for example, you may be interested in the paper Learning to learn by gradient descent by gradient descent (2016, NeurIPS).
