45

Usually I upgrade my Ubuntu installation through a ssh connection. Sometimes this ssh connection would be lost or I would accidentally close the terminal window.

It is possible to check the upgrade status after a ssh re-login into the computer?

Josema
  • 563

7 Answers7

60

The following logs are related to apt upgrades:

/var/log/apt/history.log
/var/log/apt/term.log
/var/log/dpkg.log

If the command was dist-upgrade, there are additional logs in:

/var/log/dist-upgrade

FYI, it is usually safe to just re-run the upgrade and apt will continue where it left off when the process died due to disconnection. However...

A GNU Screen Primer:

When ssh'ing into a remote server and starting a long-running process in the foreground, it is best practice to use GNU Screen. Screen provides a virtual terminal that continues running even if your ssh connection is lost.

Install screen:

sudo apt-get install screen

Run screen:

screen

After running screen you will get a command line prompt as with a normal terminal. You can then run the upgrade from inside screen:

sudo apt-get upgrade

To understand how this works, "detach" screen by pressing Ctrl+a, d. This will return you to the non-screen terminal. You can see the list of running screens with

screen -list

If you only have one screen running, you can reattach it with:

screen -raAd

(This detaches screen in case it is attached elsewhere, and reattaches it to the terminal you are currently running.)

Typically you cannot scroll 'normally' from within screen without some extra setup. To scroll within screen, press Ctrl-Esc to enter cursor mode. You can then scroll down and up with j and k. Press Esc again to exit cursor mode.

There are many more resources on the net available for additional screen functions. It is an invaluable standard tool for system administration.

See also:

Community
  • 1
doublerebel
  • 776
  • 4
    +1 voor actually answering the question AND mentioning screen :) – Nanne Dec 07 '12 at 09:04
  • 3
    Also, screen -x - attach to running screen without detaching others, making the screen session "multiplayer". – SF. Jun 14 '13 at 08:25
  • This is helpful, but includes prevention in addition to the answer. Moreover, the correct log to look at is cited, but a beginner user might not be familiar with the tail -f command and flag option, which will allow the user to observe the progress in real-time (or see that it crashed) upon "re-login." I know its old and accepted, but I think tail should be added to this instruction set, because in lacking this detail, the answer below by @TheAnonymousBear is more direct and to the point. @doublerebel – oemb1905 Jul 06 '19 at 23:47
  • Often sudo dpkg --configure -a will continue to the apt upgrade, when that was still spending. – Melroy van den Berg Jan 14 '20 at 21:08
25

I had an upgrade in progress (asking me whether to keep or replace a config file; this information I took from /var/log/apt/term.log) when my terminal was closed by another person. Restarting apt upgrade complained about a lock file being held.

My fix was to do the following:

sudo killall apt # to release the lock
sudo dpkg --configure -a # to fix the interrupted upgrade
sudo apt upgrade # to make sure no packages are left not upgraded
Ruslan
  • 1,733
22

To see real-time output from background apt job, use:

sudo tail -f /var/log/apt/term.log
anatoly techtonik
  • 2,511
TheAnonymousBear
  • 355
  • 2
    This is the correct answer - the answer above just states the location of some helpful logs then switches to prevention. This answer shows the user where to look, and how to look at it (tail) after what they called a "re-login." – oemb1905 Jul 06 '19 at 23:44
13

In addition to doublerebel's answer, I noticed an alternative today.

I went to bed last night after starting an upgrade over SSH. I stupidly forgot to start it in screen and lost my SSH session overnight.

I was just about to start researching retty when I noticed that root had started a screen session.

me@GAMMA:~$ ps aux | grep -E 'release|upgrade|apt'
root      6208  0.0  0.0  29140  1628 ?        Ss   01:57   0:05 SCREEN -e \0\0 -L -c screenrc -S ubuntu-release-upgrade-screen-window /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
root      6209  0.2  5.6 287428 93144 pts/2    Ss+  01:57   3:13 /usr/bin/python /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
root      6239  0.0  0.0  50052  1184 ?        Ss   01:58   0:00 /usr/sbin/sshd -o PidFile=/var/run/release-upgrader-sshd.pid -p 1022
root      7306  0.0  4.6 287432 77284 pts/2    S+   02:43   0:08 /usr/bin/python /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
me       26829  0.0  0.0   9440   956 pts/5    S+   22:18   0:00 grep --color=auto -E release|upgrade|apt

So I listed root's screens and attached to it:

me@GAMMA:~$ sudo screen -list
There is a screen on:
        6208.ubuntu-release-upgrade-screen-window       (12/11/2013 01:57:58 AM)        (Detached)
1 Socket in /var/run/screen/S-root.
me@GAMMA:~$ sudo screen -x -r

And Bam! I was back in the game.

Huckle
  • 7,188
  • 1
    I thought you forgot to start screen. How was screen running if you "forgot to start in screen?" – oemb1905 Jul 06 '19 at 23:54
  • 1
    @oemb1905 Because Ubuntu starts one for you, under the assumption that you'll forget :) – Huckle Jul 07 '19 at 03:15
  • interesting, is this with the do-release-upgrade command specific to ubuntu? I have never had a need to check on Debian, which I use exclusively, because I always run it manually, detach, and then come back. And, of course, we use sudo apt dist-upgrade after changing /etc/apt/sources.list instead. – oemb1905 Jul 07 '19 at 05:40
  • I found it, yes this is Ubuntu-specific so any pure Debian folks like me that poach fixes from AskUbuntu should not assume this will happen on their systems. Original thread on this topic: https://serverfault.com/questions/387547/how-do-i-reattach-to-ubuntu-servers-do-release-upgrade-process#427008 – oemb1905 Jul 07 '19 at 05:43
  • How does it know you will have screen installed? – Nacht Aug 07 '19 at 23:49
1

This post helped me to do better in the future: use 'screens'

This Debian post helped me to fix my acute problem:

https://wiki.debian.org/Teams/Dpkg/FAQ

sudo fuser -vki -TERM /var/lib/dpkg/lock /var/lib/dpkg/lock-frontend

sudo dpkg --configure --pending
TheGlasses
  • 139
1

Had exactly the same issue, lost my connection and the dpkg process was waiting for input.

Maybe next time try: sudo dpkg --configure -a

Sebastian Faujour
  • 29
  • 1
    When I try this, all I get is, "dpkg: error: dpkg frontend is locked by another process" – CivMeierFan Nov 12 '18 at 18:42
  • 1
    I did a context grep to see that thankfully it was a subprocess that generated an message dialog waiting for input, so I was able to just kill that without killing the whole apt upgrade process. – CivMeierFan Nov 12 '18 at 18:48
  • 1
    This approach ignores investigation into whether the dpkg process is still running on the system upon relogin. Moreover, if it is running, then this could be potentially harmful at worst, or just bad practice at best, since dpkg will have a lock on it in /var/dpkg/lock if it is still running. And regardless, it does not answer the question of how "to check the upgrade status" and, instead, will only work if the upgrade crashed (and only then if the lock is not active). I would not recommend this approach to anyone. Respectfully, oemb1905 – oemb1905 Jul 06 '19 at 23:52
  • Does not answer to the question – Kevin Labécot Jan 11 '22 at 14:13
  • After years of finding this Answer again and reading this comments - you're right! not the recommended way and not answering the question at all! – Sebastian Faujour Jan 28 '22 at 09:09
0

TL;DR: After losing ssh connection you might have an unreachable console dialog blocking your upgrade from continuing. Kill the whiptail process to allow it to continue.

I had a similar problem and while some of the answers were useful, they didn't help me resolve my problem. But between @Huckle's answer and @CivMeierFan's 2nd comment on @Sebastian Faujour's answer, I found my way forward.

"ps aux | grep -E 'release|upgrade|apt'" showed a whiptail process (also shown with "/var/log/apt/term.log", but in that case it just dumped an unusable console prompt. Helpful to understand why the upgrade was blocked, but not helpful for resolving the problem).

Killing the whiptail process allowed the upgrade to continue.

ajh158
  • 209