2

I have an Ubuntu 20.04.6 LTS server.

  1. It suddenly started refusing ssh connections citing /bin/bash: No such file or directory.
  2. Reboot the machine fails with a kernel panic, with the following excerpts:
Begin: Running /scripts/init-bottom ... mkdir: can't create directory '/root/lib/modules': Read-only file system
[...]
run-init: can't execute '/sbin/init', No such file or directory
[...]
run-init, can't execute '/etc/init': Permission denied
[...]
Kernel panic not syncing: Attempted to kill Init! exitcode-Ox00000100`
  1. Booting from a live USB, and looking inside the root disk. /sbin/init is not missing but it is a softlink and the /lib/systemd/systemd it points to is missing. This leads me to find out that /lib is a directory containing only a x86_64-linux-gnu folder, while in other machines, /lib is usually a softlink to usr/lib
  2. After backing up /lib and replacing it with a softlink to usr/lib, I am now able to reboot and ssh into the machine!
  3. However, when trying to run sudo apt update and sudo apt upgrade to make sure my packages are in order, I am now faced with these errors that make apt fail:
/usr/bin/python3: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/python3)

How may I recover the system in this situation, short of re-installing Ubuntu? Also, what may have caused this situation?

Edit 1:

Here is the content of the x86_64-linux-gnu I found at /lib originally. I don't think it contains as many files as it's supposed to

$ ls [...]/lib/x86_64-linux-gnu/
libexpat.so.1  libexpat.so.1.8.7  libhistory.so.8  libhistory.so.8.1  libreadline.so.8  libreadline.so.8.1

$ diff [...]/lib/x86_64-linux-gnu /lib/x86_64-linux-gnu | grep -v "Only in /lib/x86_64-linux-gnu"
Binary files [...]/lib/x86_64-linux-gnu/libexpat.so.1 and /lib/x86_64-linux-gnu/libexpat.so.1 differ
Only in [...]/lib/x86_64-linux-gnu: libexpat.so.1.8.7
Binary files [...]/lib/x86_64-linux-gnu/libhistory.so.8 and /lib/x86_64-linux-gnu/libhistory.so.8 differ
Only in [...]/lib/x86_64-linux-gnu: libhistory.so.8.1
Binary files [...]/lib/x86_64-linux-gnu/libreadline.so.8 and /lib/x86_64-linux-gnu/libreadline.so.8 differ
Only in [...]/lib/x86_64-linux-gnu: libreadline.so.8.1

Also, these warnings also show up just when I SSH in:

/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/bin/xauth)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/xauth)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libX11.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libX11.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libXau.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libbsd.so.0)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libmd.so.0)
LemmeTestThat
  • 241
  • 1
  • 2
  • 8
  • 1
    For that one, try copying the directory x86_64-linux-gnu that you backed up from /lib/x86_64-linux-gnu to /usr/lib/x86_64-linux-gnu assuming you have previously moved it from its original location and it's still intact ... You seem to have messed up your filesystem hierarchy quiet a bit. – Raffa Jul 20 '23 at 11:22
  • 2
    AFAIK Ubuntu 20.04 uses libc version 2.31 - so it suggests your /usr/bin/python3 is not the default version from the Ubuntu repo? – steeldriver Jul 20 '23 at 11:24
  • @Raffa, I just edited the question to add more details on the contents of the x86_64-linux-gnu I backed up. I don't think it has everything it's supposed to, and the few packages still in it don't seem to relate to the warnings. – LemmeTestThat Jul 21 '23 at 02:19
  • @steeldriver I think you're right. Any way I could downgrade all packages to their default versions in Ubuntu repo, or install a higher version of libc & libm to match what is now being required? All without being able to use apt? – LemmeTestThat Jul 21 '23 at 02:23
  • @steeldriver, by the way, I can see /usr/bin/python3 -> python3.10 and there is a python3.8 available there. Maybe I could try redirecting /usr/bin/python3 to 3.8? – LemmeTestThat Jul 21 '23 at 02:32
  • This related post might help: Terminal and other apps suddenly won't open – Raffa Jul 21 '23 at 07:30
  • @LemmeTestThat yes re-linking /usr/bin/python3 to python3.8 should help – steeldriver Jul 21 '23 at 11:33
  • @steeldriver Linking python3.8 did seem to get apt going further, but then It still crashed with an error from /lib/x86_64-linux-gnu/libsqlite3.so.0 – LemmeTestThat Jul 24 '23 at 02:58

0 Answers0