Restore specific kernel & modules (nvidia drivers)

Question

Background: I've been running Ubuntu for years (started at 16.04, now at 20.04), and been constantly fighting with NVIDA drivers which I need as I use CUDA. As recently as yesterday, my NVIDIA 460 drivers were working fine, and an apt upgrade broke them again: Ubuntu 20.4 update broke my Nvidia 460 driver config

What I want to achieve:

Create a restore point of a kernel and its modules (eg nvidia drivers) I'm happy with
Whenever NVIDIA drivers break (or something else breaks badly), restore it

What I already have:

GRUB which seems to allow choosing specific kernels to boot from

(screenshots for illustration, not reflecting latest version)

Questions:

Restore point containing kernel+modules: is it possible to create (if so how), or am I simply misunderstanding how kernels & modules are managed on linux (ie a kernel wouldn't include the nvidia drivers) ?
Restoring from grub: It seems /etc/grub.d/40_custom is the file I want to modify as it's designed specifically for custom menu entries. Do you confirm this is the intended way of booting custom kernels or should I be looking at another file?

/etc/grub.d/40_custom:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

sancho.s ReinstateMonicaCellio · Accepted Answer · 2021-03-19T14:26:40.640

4

The package for the linux kernel is separate from the nvidia drivers. But each kernel version has its corresponding drivers. If you had the nvidia drivers installed for, say, 4.10.0-28, then booting that kernel should use the corresponding drivers.

Other modules may be provided by different packages, e.g., linux-modules-5.8.0-45-generic, linux-modules-extra-5.8.0-45-generic.

So you should already have your "restore point"s, and you can use them via grub, as you show. As for the custom entries in grub, you don't need that for older kernel versions, those are added automatically when updating. You might need that if you want other customization, though.

It is not clear if that is not enough for you, or if that did not work as suitable restore points. Giving a specific example of what did not work for you, if you have any, may help clarifying.

edited Mar 19 '21 at 14:26

answered Mar 19 '21 at 14:20

sancho.s ReinstateMonicaCellio

14,674
11
44
97

I guess the challenges are: knowing which module contains which packages, making sure old kernels don't get deleted and knowing which version to restore (since with apt update new versions can be created without me noticing, and maybe I realize something was broken only several versions later). The lack of answers on my question + challenges make me realize that maybe I'd rather use a imaging tool like Clonezilla to fully control what I backup/restore: https://clonezilla.org/screenshots/?in_path=/00_Clonezilla – Max Mar 19 '21 at 14:34
@Max - The "challenges" you pose here look to me like a different issue from what you asked, a question on their own. I will try answering them here, but I suggest using the (edited) question as the place to add them.
1. Which module contains which packages. See https://askubuntu.com/a/1259815/226614 for an example on how to go from module to package (and how it solved a specific problem).
2. Making sure old kernels don't get deleted. See https://help.ubuntu.com/community/RemoveOldKernels , depending on your configuration, they might be automatically removed or not...
– sancho.s ReinstateMonicaCellio Mar 19 '21 at 16:19
Knowing which version to restore. As said, there is nothing to restore, it is not like Windows Restore point. You simply boot the version you want.
The lack of answers... Well, I guess we are commenting on an answer. I don't know how many answers you consider a satisfactory number.
Clonezilla/imaging. That would work ok for many cases, probably including yours. But from what I understood, it would be an overkill for you.

Again, if you post a specific example of a case you could not work, that would help.

sancho.s ReinstateMonicaCellio

Mar 19 '21 at 16:23

Thanks Sancho, you're most certainly right. The challenge with Stackexchange is that it forces you to narrow down possibilities to specific questions which might limit you in your search for solutions. Afterthought: the mere fact that kernels can be purged and reside on the OS partition should deter me from using kernels as means of backup/restore. I think disk imaging, although overkill in the sense it might restore unnecessary data, seems considerably simpler/safer (just image the disk to external storage and restore when needed). Nonetheless thx for your answers, I learned a lot. – Max Mar 21 '21 at 07:14

WinEunuuchs2Unix · Answer 2 · 2021-03-22T18:03:25.673

1

For myself I stay on the same nVidia driver and kernel chain:

$ uname -r
4.14.216-0414216-generic
$ nvidia-smi -q
==============NVSMI LOG==============
Timestamp                           : Mon Mar 22 11:49:28 2021
Driver Version                      : 384.130
Attached GPUs                       : 1
GPU 00000000:01:00.0
    Product Name                    : GeForce GTX 970M
    Product Brand                   : GeForce
    Display Mode                    : Enabled
    Display Active                  : Enabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-30fab9bc-fe6f-ec05-e8e6-c151a1a96121
    Minor Number                    : 0
    VBIOS Version                   : 84.04.79.00.0A

There is no need to update kernel chain until hardware newer than the kernel chain is installed. The disadvantage is you need to occasionally update the kernel (on the same chain) if your system crashes and you don't know why.

Kernel 4.14 is an LTS (Long Term Support) kernel that is updated for five years with security and bug fixes. Updates are made by the Linux Kernel team and published by the Ubuntu Kernel team.

It's not what I would call a "custom" kernel but many will call it such. When you upgrade within the same kernel chain, say 4.14.188 to 4.14.216 like I did last month grub doesn't automatically make it the new default on the main menu:

So after kernel update go into "Advanced options" and select the new kernel. Make sure you configure grub to always boot with the last used option:

Is there a way for grub to automatically reboot into Windows from Windows?

Although I have a theme on my grub menu, the menu options stay the same in "regular" grub.

My basic rule of thumb is "If it ain't broke don't fix it". I discovered this after many times of repairing things after software upgrades.

Something to keep in mind when you do upgrade your drivers, kernel and firmware. Do it all at once, test thoroughly and then freeze the setup for many months or even years.

edited Mar 22 '21 at 18:03

answered Mar 22 '21 at 17:54

WinEunuuchs2Unix

102,282

"For myself I stay on the same nVidia driver and kernel chain" I'd be happy with that, but as I do working on coding/machine learning, I'm constantly adding/updating various packages, and by default apt update seems to update everything, which can break NVIDIA drivers, happened again very recently: https://askubuntu.com/questions/1323872/ubuntu-20-4-update-broke-my-nvidia-460-driver-config#comment2256793_1323872. How can prevent Ubuntu from updating the core kernel and NVIDIA drivers while still allowing updates for the rest? – Max Mar 23 '21 at 07:57
@MAX Hidden in the answer above is that when I upgrade, Ubuntu does upgrade the core kernel chain but I don't use that chain. Kernels Ubuntu upgrades are kept for two generations and then deleted when autoremove is run. I use the LTS kernel chain which doesn't have radical changes, just security updates and bug fixes. As far as nVidia goes it is setup in additional drivers section and the one driver version is always flagged as the one to use, never the most current which I believe gets automatically updated. I also do software development in python so a stable environment is needed. – WinEunuuchs2Unix Mar 23 '21 at 23:18
about "I use the LTS kernel chain which doesn't have radical changes, just security updates and bug fixes": I also use LTS, that is 20.04 LTS, and 2 weeks ago a simple apt-get update broke my nvidia drivers, and as you saw on https://askubuntu.com/questions/1323872/ubuntu-20-4-update-broke-my-nvidia-460-driver-config#comment2256793_1323872 I'm not the only one this happened to. The idea of freezing the setup for years isn't an option either (what if there are security patches?). I think this thread reinforces my idea that disk imaging is what I need to be in peace – Max Mar 24 '21 at 08:18
@Max I'm using LTS kernel 4.14 I wouldn't call the kernel that comes with 20.04 the same thing. I'm not sure of the version but Ubuntu LTS is not the same thing as Kernel LTS. For example 5.4 and 5.10 are LTS: https://www.zdnet.com/article/linux-5-10-will-be-the-next-long-term-support-linux-kernel/ To make a long story short when I get new hardware say USB4 I'll upgrade to the newest LTS kernel that supports it and stay there. – WinEunuuchs2Unix Mar 24 '21 at 11:20
I see, thanks for clarifying. I think I don't have the linux skills (and no time to allocate to learning, already plenty to do) to start dealing with kernels. Asking question here was great as it gave me confirmation that for me I should go down the disk imaging path – Max Mar 24 '21 at 11:28
@MAX installing a Mainline Linux kernel from Ubuntu is straightforward. It only takes a few minutes to download, install and reboot: https://askubuntu.com/a/885165/307523. There is no extra skills other than sudo dpkg -i instead of sudo apt install. I think imaging is a waste of time and gives less flexibility, eg inability to instantly reboot to a different kernel version if need be. – WinEunuuchs2Unix Mar 24 '21 at 23:21
I think from that point on it becomes more subjective (eg about what each person is used to doing and thus how much time/learning it will require) – Max Mar 25 '21 at 11:15

Restore specific kernel & modules (nvidia drivers)

2 Answers2

Linked