1

I removed Windows 10 and installed Ubuntu 18.04 LTS in UEFI Secure Boot Mode in 2019 on my development machine, aside from initial complications of getting the UEFI system working, I've had no problems. I let the kernel update up to 2020 it was upgraded fron 4.18 to 5.4.0-47.
What happened yesterday was that I was working on a project, I connected my machine to the internet, something I hadn't done since mid 2020 and I left my machine for a while and the screen went black, it wasnt the screen saver, I always had the power options to prevent hibernation, so can't have been that, it could have been a power surge as I realised I wasnt plugged into my surge protector, the one time I was plugged directly into a wall socket which appeared to buzz for a few seconds, the machine was unresponsive so I held the power button down to turn it of and restart it, but it would no longer boot, however luckily the drive and all its linux file systems appear to be intact after checking them in emergency mode, but whenever I turn on the machine it boots into grub and allows me to boot normally or into a previous kernel, but then it always boots into emergency mode and I am unsure how to proceeed and dont know what the cause was as there were no logs written on the day of the crash,as it stands I am having to consider backing up all my data and reinstalling, perhaps a later ubuntu version or another distro?

Here are the lines from journalctl -xb that allude to any problems:

Aug 03 15:19:22 DEMO kernel: [Firmware Bug]: TPM Final Events table missing or invalid
Aug 03 15:19:22 DEMO kernel: secureboot: Secure boot enabled<br /> 
...
Aug 03 15:19:22 DEMO kernel: Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7
Aug 03 15:19:22 DEMO kernel: Lockdown: systemd: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7 
..
Aug 03 15:19:27 DEMO systemd[1]: Started Flush Journal to Persistent Storage.
Aug 03 15:20:52 DEMO systemd[1]: dev-disk-by\x2duuid-D001\x2d5175.device: Job dev-disk-by\x2duuid-D001\x2d5175.device/start timed out.
Aug 03 15:20:52 DEMO systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-D001\x2d5175.device.
Aug 03 15:20:52 DEMO systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/D001-5175.
Aug 03 15:20:52 DEMO systemd[1]: Dependency failed for /boot/efi.
Aug 03 15:20:52 DEMO systemd[1]: Dependency failed for Local File Systems. 
Aug 03 15:20:52 DEMO systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency' 
Aug 03 15:20:52 DEMO systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
Aug 03 15:20:52 DEMO systemd[1]: boot-efi.mount: Job boot-efi.mount/start failed with result 'dependency'.
Aug 03 15:20:52 DEMO systemd[1]: systemd-fsck@dev-disk-by\x2duuid-D001\x2d5175.service: Job systemd-fsck@dev-disk-by\x2duuid-D001\x2d5175.service/start failed with result 'dependency'.
Aug 03 15:20:52 DEMO systemd[1]: dev-disk-by\x2duuid-D001\x2d5175.device: Job dev-disk-by\x2duuid-D001\x2d5175.device/start failed with result 'timeout'.`

etc/fstab reads:

UUID=MYUUID / ext4    errors=remount-ro 0       1 
\#/boot/efi was on /dev/sda1 during installation  
UUID=D001-5175  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0

EDIT
The cause and effect of this problem are practically identical to:
EFI Lockdown, Can't Boot Ubuntu 16.04 or Windows 10
However the solution given for that is windows based, but in my case my system is not a dual boot and I am only running ubuntu and surely there is an ubuntu based solution?

This is what I see when I try to boot my system normally:

[   0.964483] Integrity: Problem loading X.509 certificate -65                                  
[   0.964494] Integrity: Problem loading X.509 certificate -65                                  
dev/sda2: clean, 539930/61022208 files, 37539912/244059136 blocks                             
[          *] A start job is running for dev-disk-by<x2duuid-D001<x2d5175.device (1min 30 / 1min 30s)
[   TIME    ] Timed out waiting for device  dev-disk-by<x2duuid-D001\x2d5175.device.                   
[   DEPEND  ] Dependency failed for File System Check on dev-disk-by-uuid/D001-5175.      
[   DEPEND  ] Dependency failed for /boot/efi.                                                   
[   DEPEND  ] Dependency failed for Local File Systems.                                         
              Starting Enable support for additional executable binary formats...                 
[     OK    ] Reached target Login Prompts.                                                     
[     OK    ] Reached target Timers.                                                            
[     OK    ] Started Emergency Shell.                                                          
[     OK    ] Reached target Emergency Mode.                                                                                                         
              Starting Create Volatile Files and Directories...                                   
[     OK    ] Reached target Sockets.                                                           
[     OK    ] Reached target Paths.                                                             
              Mounting Arbitrary Executable File Formats File Systems...                                      
[     OK    ] Mounted Arbitrary Executable File Formats File Systems.                                 
[     OK    ] Started Enable support for additional executable binary formats.                      
[     OK    ] Started Create Volatile Files and Directories.                                    
              Starting Network Time Synchronization...                                            
              Starting Update UTMP about System Boot/Shutdown...                                  
              Starting Network Name Resolution...                                                  
[     OK    ] Started Update UTMP about System Boot/Shutdown.                                          
              Starting Update UTMP about System Runlevel Changes...                                            
[     OK    ] Started Update UTMP about System Runlevel Changes.
[     OK    ] Started Network Time Synchronization.
[     OK    ] Started Network Name Resolution.
[     OK    ] Reached target Host and Network Name Lookups.
[     OK    ] Reached target System Time Synchronized.
You are in emergency mode.  After logging in , type “journalctl -xb” to view system logs, “systemctl reboot” to reboot, “systemctl default or “exit” 
to boot into default mode.  Press Enter for maintenance
(or press Control-D to continue):

EDIT As galexite has proven in the comments, this problem is not a Secure Boot or TPM issue. To try and determine the actual cause, I started going through all the logs and found a crash log, (see pastebin.ubuntu.com).

What I also did before this crash, which I didnt think would be relevant or significant was that in Ubuntu Software Center, I installed Firefox and some music writing apps and in the crash log it appears that it either triggered a software update or some background proceess had already started in the background of its own accord, that set about removing an alarming amount of essential packages, it even tried to remove the sudo package only the request was rejected and all the package removals in this crash log state that I have asked that the package be removed, when I only wanted to install a few packages and not remove anything at all, its not like I installed them manually myself and I had deliberately prevented Software Updates, prior to the kernel upgrade.

I have put the following output into the Pastebin from $service --status-all, showing there is only 1 service running and I can only assume its because the software update either removed, half-installed or half-configured many of the software packages.

EDIT Note: That despite me suspecting the cause was possible a power surge, it was only an assumption as I cannot be sure, there is no hardware damage, but there appears to have been a major software failure, what I can be sure of is that I saw the screen go off which appeared to be the system power management settings kicking in, I left my machine for quite some time, I was sure I had set the option to stop the system from going into hibernation mode, but that was just after installation some years back, because hibernation had always caused problems for me and in 18.04 apparently there was a bug identified were a system could be compromised if it went into hibernation mode, so I was right to do that, however I cannot now be sure my power settings were preserved after all all the updates and forced unattended-upgrades. I can be sure that once I had recovered the system merely to the point whereby I could login via text only mode and I thought I could simply check all the essential services are running, especially networking and the firewall, so I could potentially complete the software updates/unatttended upgrades, which I have found at the time of the crash, were underway, but incomplete and as I found the logs state that many packages are half installed or half configured or removed completely. So in text mode, I ran the following command to determine what services were running:

service --status-all |nl

     1   [ + ]  acpid
     2   [ - ]  alsa-utils
     3   [ + ]  anacron
     4   [ - ]  apparmor
     5   [ - ]  apport
     6   [ - ]  avahi-daemon
     7   [ + ]  binfmt-support
     8   [ - ]  bluetooth
     9   [ - ]  console-setup.sh
    10   [ - ]  cron
    11   [ - ]  cups
    12   [ - ]  cups-browsed
    13   [ - ]  dbus
    14   [ - ]  dns-clean
    15   [ - ]  gdm3
    16   [ - ]  grub-common
    17   [ - ]  hwclock.sh
    18   [ + ]  irqbalance
    19   [ + ]  kerneloops
    20   [ - ]  keyboard-setup.sh
    21   [ + ]  kmod
    22   [ - ]  network-manager
    23   [ - ]  networking
    24   [ - ]  plymouth
    25   [ - ]  plymouth-log
    26   [ - ]  postfix
    27   [ - ]  pppd-dns
    28   [ + ]  procps
    29   [ - ]  rsync
    30   [ - ]  rsyslog
    31   [ + ]  saned
    32   [ - ]  speech-dispatcher
    33   [ - ]  spice-vdagent
    34   [ - ]  udev
    35   [ - ]  ufw
    36   [ + ]  unattended-upgrades
    37   [ - ]  uuidd
    38   [ + ]  whoopsie
    39   [ - ]  x11-common

So the objective was clear, now I was able to at least run the system in text mode, I needed to start the essential services, starting with the network manager and the firewall, since I could then allow the software updates/upgrades to complete and then recover the system sufficiently so I could resume a normal boot, but when I tried to start the network-manager.service nothing happened, no output from the command, or anything.

When I tried to start ufw: Failed to restart ufw.service: Unit ufw.service is masked. When I tried the command to unmask the service it said Removing ufw...

I tried to unmask and restart the rest of the services in turn and either the commands returned no output or nothing happened.

Consequently I appear to be stuck in a catch 22 situation, I cannot recover this system, because I cannot start essential services, that will enable any recovery, or any services for that matter as in this emergency state Ubuntu has masked all of the services that are not running and I have to ask what is the point of emergency mode if you cannot recover from it? I also cannot understand why all these services are masked in the first place and what the rationale is for that is? and why any attempt to unmask even with sudo privileges is either ignored or silently doesn't work or the service is removed completely?

  • Secure boot isn’t your problem I’m afraid, the kernel is informing you it will only trust signed modules, restrict access to capabilities which allow programs to interfere with kernel memory and so on. The TPM isn’t required to boot. However, the logs before all the disk-related timeouts may be useful. There is an issue accessing the disk. – galexite Aug 03 '21 at 20:52
  • 3
    My guess is that unattended-upgrades was trying to perform a year and a half worth of updates classified as critical bugfixes or critical security patches and when you forced shutdown you left your system in a broken state. Can you drop to a root shell prompt to run apt -- fix-broken install? – Nmath Aug 03 '21 at 20:55
  • @Steve, can you post the rest of your logs, that you have cut? Can you surround your logs in three backticks (`) or tildes (~) on either side, as shown in the formatting guide? – galexite Aug 03 '21 at 21:39
  • @galexite I can do but just wanted to avoid pasting in all logs as they are huge. If you let me know which parts of which logs will be of interest, please specify and I'll add them. Also thanks for letting me know how to best format them, its been so long since I last posted I'd forgotten. –  Aug 03 '21 at 21:57
  • You can use https://pastebin.ubuntu.com if your pastes are too large – Nmath Aug 03 '21 at 22:08
  • @Nmath, Thanks. I spoke to soon when I replied to your first comment, you were right about the unattended-upgrades, I had only checked all the log entries from 3rd Aug 2021, but on the 2nd my system started updates, I was only onlne for 10 mins and it performed tons of updates in that time and I found loads of log entries with "half unstalled" and "half configured", on the 3rd it may have tried to resume the updates, so when this power surge occurred, it likely put my system in an indeterminate state causing the kernel to protect itself and initiate lockdown. –  Aug 03 '21 at 22:43
  • @Steve, all systems with Secure Boot display this log message. However, the system boots fine. See man kernel_lockdown.7. You upgraded to a Linux kernel which supports lockdown. If lockdown did interrupt a process, then the kernel reports any breach as a new message, see that manual page. The signatures are integrated within the Linux kernel. The OS does not need to access the TPM to perform secure boot. – galexite Aug 04 '21 at 06:33
  • @galexite. You are right about Secure Boot not being the problem. I looked through the logs from the previous month and found the same entries including: kernel: [Firmware Bug]: TPM Final Events table missing or invalid and Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7 and my system was runnng normally back then and this proves both of your points to be correct. So it looks like I'll have to edit the question title, only problem is I have no idea what to, as I dont know what the problem is now. .. –  Aug 05 '21 at 16:42
  • @Steve you have the identical issue to another question I’m trying to solve. The EFI System Partition is taking too long to mount. Can you please try commenting out the /boot/efi entry in /etc/fstab from either the emergency shell or a live DVD or USB, then try booting? – galexite Aug 05 '21 at 18:31
  • Also, can you verify that D001-5175 is the correct UUID for your EFI System Partition using blkid? – galexite Aug 05 '21 at 18:34
  • "galaxite. Cool, I was starting to think about giving up as I saw another question, with the same cause and effect, however that was a dual boot scenario with windows and the qp fixed it himself with windows utils. I already ran blkid to check and if I recall the UUID was a match. I will try commenting out the /boot/efi entry now. I'll get back to you with the results –  Aug 05 '21 at 19:20
  • @galaxite. I made the suggested edit to /etc/fstab and hey presto I am out of emergency mode and can now login to my system but only in text mode atm, I am now trying work out how to start my system in gui mode. Thanks for your help, if you have anything else you want me to try then I'm all ears. –  Aug 05 '21 at 19:46
  • I would recommend updating your system using sudo apt update && sudo apt full-upgrade, possibly to resolve any missing dependencies. You might get an error from apt whilst installing those updates, though we can resolve them as they come. – galexite Aug 06 '21 at 12:23
  • "galexite. Thanks for your time in helping me resolve this. I have no doubt that your answer will work, the only problem is that I dont have any networking services atm, I have a system but it looks like the bare bones and unless I can get it connected I cant implement your solution. I seem to be stuck in a catch-22 situation atm. –  Aug 06 '21 at 13:03
  • @Steve, it will at least be possible to complete the upgrade of the downloaded packages, because they will be in apt's cache. apt downloads all the packages before installing them. – galexite Aug 06 '21 at 17:26
  • @Steve, follow my answer, but ignore the steps relating to debsums and apt update, you should be able to even run apt upgrade. – galexite Aug 06 '21 at 17:27
  • @Steve, if you still want help restoring the system, I can walk you through installing the missing packages from an offline repository or USB drive over chat if you'd like. – galexite Aug 09 '21 at 16:30
  • @galexite. Apologies, I went offline soon as i posted the last message. Well there's not really any point for me now, as I have wiped that system and busy configuring the new system, but if you could add those steps to your answer it would be helpful should anyone else ever have the same or similar problem. I am thankful though despite the crash and my frustration, that all my data was preserved and untouched, testament to what an exceptional, well designed, maintained and robust operating system linux actually is. –  Aug 09 '21 at 18:53
  • @Steve, no filesystem damage was done regardless of what OS was running. Some packages essential for the GUI got removed, and there is a bug somewhere mounting ESPs... – galexite Aug 09 '21 at 21:02
  • @galexite, sorry I dont quite get what you saying "No damage was done... ?", in reference to my filesystem and OS, it was left inoperable and irreperable, despite dedicating nearly a week of my time and in that time nobody could provide a workable solution. In reference to other part, what your saying is that my filesystem would in theory not sufferred any damage regardless of the OS? which appears to imply every OS in the world is perfect and handles crashes without any filesystem damage? in reality this OS never even recorded a single log of the event, no log, segv info nothing. –  Aug 10 '21 at 11:17

1 Answers1

1

You should check the integrity of installed packages and complete the upgrade to restore your system back to normal.

Ensure all unpacked packages have been configured:

sudo dpkg --configure -a

The debsum package achieves integrity verification using each package's MD5SUMS file. You can install it:

sudo apt update
sudo apt install debsums

You then need to clear apt's cache, and initialise debsums by downloading the packages again:

sudo apt clean
sudo debsums_init

Then run a manual check, whereby debsums reports changed or corrupt files:

sudo debsums -cs

Check to make sure the configuration changes are ones that you expect. If any errors are reported, you need to re-install those packages:

sudo apt install --reinstall broken-package

You should then complete the update.

Before you perform your update, however, as we have removed the /boot/efi mount from /etc/fstab, you should mount it again in case GRUB needs updating. Uncomment the line in /etc/fstab, then run:

sudo mount /boot/efi

You can then proceed with the update:

sudo apt update
sudo apt upgrade --with-new-pkgs

Reboot. If the system does not come up again because the ESP is still not mounting, then more troubleshooting needs to be done.

galexite
  • 576
  • 2
  • 7
  • Thanks. I followed your answer as suggested. The update said all packages were up to date, the upgrade said 0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade. After I completed the steps, I ended up at back in emergency mode with exactly the same set of messages output. I put the results from the update/upgrade in the pastebin. –  Aug 06 '21 at 19:51
  • Thanks for all your help, I would have preferred to persevere and try to restore my system, but to acheive that would involve identifying the source of the problem, but with no logs from the event it's futile and I dont have time for a full forensic analysis. I've spent nearly a week on it now and couldn't justify dedicating any more precious time on it so I reinstalled with 20.04 LTS, which took no time at all and I am impressed by all the improvements and from now I am going to image my system regularly and if anything like that happens I can restore it from backup system in minutes. –  Aug 09 '21 at 16:52
  • All good, no problem. The issue is caused due to a bug I believe mounting /boot/efi, and some packages required for the GUI are missing. – galexite Aug 09 '21 at 16:52
  • Oh right, good work, thanks for your help, your a good egg. At least if you have uncovered a bug looking at this problem its justified the time we spent on it and the issue will never re-occur :) –  Aug 09 '21 at 16:59