While training an online model, usually we use progressive validation https://maxhalford.github.io/blog/online-learning-evaluation/. My question is what we should do when we detected performance regression. In a traditional offline training setting, we won't promote a model unless it surpasses previous models in validation which is not possible in online setting.
Related to this question, is there a way to version models in online setting? Maybe we just keep a snapshot of the model on a daily basis?
