62

So I ran into that weird "packages being held back" problem while updating my system using apt. After some research, it appears to be the result of "phased updates".

  • What are phased updates, anyway?
  • Why does Ubuntu use them?
  • How do they benefit me?
  • Can I (and should I) turn them off or work around them?
  • Why are apt's messages regarding phased updates so unclear?
ArrayBolt3
  • 3,129

2 Answers2

85

What are phased updates?

Phased updates are software updates that are gradually rolled out to users rather than all users getting the updates at the same time. In an update system that doesn't use phased updates, all users are immediately offered an update as soon as it is made available. When phased updates are used, only some users get an update when it is first released. As time passes, more and more users get the update, until the update is finally "fully phased" and everyone is offered the update.

Ubuntu has been using a phased update system for Ubuntu Desktop since Ubuntu 13.04 (almost ten years ago!), however, up until now, only the GUI updater for Ubuntu was able to handle phased updates. Updating via apt would ignore update phasing. However, starting with Ubuntu 21.04, apt also uses phased updates, allowing phased updates to apply to all versions of Ubuntu, including Ubuntu Server and Ubuntu running in containers. Incidentally, this change also resulted in the now-infamous "The following packages have been kept back" message from apt during many system updates. While phased updates in apt have been a thing since Ubuntu 21.04, they have come to light most clearly in Ubuntu 22.04 LTS.

Why does Ubuntu use them?

Occasionally, just like anyone else, Ubuntu developers make mistakes. While updates are tested before they are released, sometimes unexpected corner cases aren't properly taken into account. And if those unexpected corner cases turn out to be somewhat common, they can start causing users problems on their systems.

In the past, before phased updates were used with Apt, any time an update was pushed out, all Ubuntu users would immediately be able to install the update. And if that update happened to be wonky, it could mess up a lot of people's systems all at once. Phased updates make it so that, in the event an update is bad, it gets caught early before everyone gets the update. Once an update is pushed out, some users get it sooner than others. And if those users' systems start reporting back crashes (or if severe problems with a package come to light), the phasing process can be stopped, making it so that the update stops being sent to users by default. This improves the safety and stability of Ubuntu.

The phasing system makes it so that different sets of users are chosen to be the first to get the updates so that there isn't one group of unlucky people who always get the updates soon after release.

How do they benefit me?

If you've encountered times when an update broke your system in the past, you shouldn't be as likely to experience that problem in the future. Update phasing makes it so that breakage is more likely to be detected early on, avoiding causing problems to users' desktops, servers, and other Ubuntu-powered devices. This directly benefits Ubuntu users by increasing the stability and reliability of Ubuntu.

Can I (and should I) turn them off or work around them?

That depends on how stable you need your system to be. If you just want to avoid any notices about packages being held back during apt updates, and you're willing to be one of the first people to get updates whenever they're released, you can turn off phased updates. This is somewhat of a risk since it means if an update is bad, you will pretty much always be the first person to get the update (essentially you're making yourself a guinea pig for the early update releases!). But it will get rid of the notices about held-back packages in apt.

If that sounds like a bad idea, leave phased updates on.

Working around phased updates by running apt install on held-back packages is probably always a bad idea - it won't permanently "fix" the held-back package notice (which doesn't actually need fixing in the first place), and it will make you install packages early. If you've already done it, it probably won't mess up your system (unless the update has a problem), but it's likely not something you want to make a habit of doing.

If you are absolutely sure that you want to disable phased updates, you can change apt's configuration by creating a file in /etc/apt/apt.conf.d, containing the following lines:

Update-Manager::Always-Include-Phased-Updates;
APT::Get::Always-Include-Phased-Updates;

You can name the file 99-Phased-Updates if /etc/apt/apt.conf.d/99-Phased-Updates doesn't already exist. Again, only do this if you really know what you're doing and are absolutely sure you need to do this (for instance, if you intentionally are installing all the latest packages since you want to help test them and can afford it if your system breaks). If you're doing this for almost any other reason than this example, there's probably a better way.

Why are apt's messages regarding phased updates so unclear?

It's actually somewhat difficult to get apt to recognize phased updates as being held back because they're phased updates, due to how dependencies work. However, we do recognize that this is an annoying problem, and it's on our radar. There's a bug report about it here: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1988819 If you have an Ubuntu SSO account and can log into Launchpad, you can click on the "This bug affects X people. Does this bug affect you?" link near the top of the page, and then click "Yes, it affects me" to increase the bug heat rating, making it more significant.

(Note: Please don't leave mean and nasty comments at the bottom of the bug report - all it does is make us annoyed or upset and doesn't actually push us to fix the bug any faster.)

Rohit Gupta
  • 348
ArrayBolt3
  • 3,129
  • 13
    Excellent and comprehensive answer. "running apt install to work around": a nasty side effect of doing this is that you are marking these packages as "manually installed" so they never will be automatically removed if all packages that need them are uninstalled. – vanadium Sep 23 '22 at 16:39
  • 1
    Great answer. I would add that you can also "turn off phased updates" by telling apt to never install updates that are being phased, which might keep your system slightly more stable, but if everyone did it then it would defeat the purpose of phased updates. (https://askubuntu.com/a/1246984) – pizzapants184 Sep 24 '22 at 07:32
  • 1
    I'm a bit surprised--was controlling server load not a concern? If updates are released to everyone all at once, you'd expect the servers providing the updates would essentially get DDoSed by millions of users and auto-update systems updating all at once. – Hearth Sep 24 '22 at 19:37
  • 4
    I assume security updates are never phased? – OrangeDog Sep 25 '22 at 10:33
  • 2
    @OrangeDog True. – ArrayBolt3 Sep 25 '22 at 10:33
  • 1
    @Hearth: From everything I've seen, the primary goal of phasing was for user system stability. I find it easy to imagine that server load may have been part of the concern, but nothing official that I've read states that, that I can remember. If it was a concern at all, it was secondary. – ArrayBolt3 Sep 25 '22 at 10:34
  • @ArrayBolt3 I would also mention that if problems are caused by outdated package then phased updates make the problem worse (I think that I proposed such edit but I do not see it in history, maybe it was rejected or I failed to submit it) – reducing activity Sep 30 '22 at 15:04
  • @reducingactivity I wasn't quite sure what it was you were saying when I saw the edit. It looked like it was saying that it may delay package updates and that could cause a problem, but I don't see how an ~2 day delay would cause any practical problems. Maybe I was misunderstanding? – ArrayBolt3 Sep 30 '22 at 17:09
  • @ArrayBolt3 if there is a bug in package that is fixed in update (or new feature) then it means that user is using inferior version for ~2 days longer. – reducing activity Sep 30 '22 at 17:15
  • @reducingactivity Ah, I get it. I guess that's true, but as far as I can tell, a couple of days delay isn't going to really be something a user would notice in their day-to-day usage of Ubuntu. Still, I see your point, and I'll try to fit it in somewhere. – ArrayBolt3 Sep 30 '22 at 17:31
  • Thanks for the detailed explanation. I have by now done a number of manual installs of the held back packages; simply because that was the method proposed throughout the last 15 years.
    1. How can I revert to the stable situation?
    2. I do find this inconvenient, maybe because of what I did for the last 15 years. I'd rather see only 'held back' packages for individual situations on my specific machine, and just not at all see those of phased updates.
     – udippel Feb 16 '23 at 12:21
  • @udippel If your system still works, there's no need to "undo" the use of "sudo apt install " to get around the warning. It's extremely rare for a package update to be pushed and then stopped, and if that happened I am very sure that the developers would be pushing an update to fix the busted one rather than leaving it there. The only problem using "sudo apt install" for this will cause is that it will make it harder to autoremove the packages in question in the future should you want to. Which probably isn't that big of a deal. – ArrayBolt3 Feb 16 '23 at 16:23
  • @udippel Sadly, the only good way to get rid of the notice about held-back packages (that I know of) is to disable update phasing entirely. The directions on how to do so are in the original answer, but obviously that comes with disadvantages. I agree that this is frustrating, and hope that eventually apt will be able to tell the difference between "phased-and-held-back" and "broken-and-held-back" packages easily. – ArrayBolt3 Feb 16 '23 at 16:26
  • 1
    Absolutely, thanks to you, ArrayBolt3. The problem might not be so visible to newcomers, but "The following packages will be upgraded: base-files 1 upgraded, 0 newly installed, 0 to remove and 6 not upgraded." until a few months ago was suppoed to give me the creeps; and invite to file a bug report. Yes, it is unfortunate, that out of the blue this is expected behaviour. I'd actually appreciate a switch, so that either the phased updates just don't pop up (default; historical bahaviour); or ARE installed; or show up as they do like recently introduced. – udippel Feb 17 '23 at 18:31
  • I'm here for the same reason as udippel. Held back packages that show up have traditionally indicated a problem in apt. Having a random subset of machines participate in any phased release is a weird decision for LTS environments, especially servers. I prefer certain machines running LTS builds that I manage to always be among the last group for most package deployments with an exception for propagated CVE fixes where I'd want them to always be in an earlier group. The current configuration options don't appear to support that model. – CubicleSoft Apr 29 '23 at 14:23
  • 1
    Here's the command that upgrades including phased updates: sudo apt -o APT::Get::Always-Include-Phased-Updates=true upgrade – Flimm Oct 13 '23 at 12:37
6

Also worth adding, as explained in the apt-preferences docs, that Phased Updates depend upon each machine's Machine ID, which maybe found (and potentially altered/copied - but generally one should avoid changing it as mentioned by @Daniel) in /etc/machine-id. So if two machines have the same machine-id they're at the same "Phase" and get the same updates.

Pierz
  • 3,063
  • 1
  • 25
  • 15
  • This is a bad idea. Your link said "anything that was recording machine-specific state with the hostname as unique identifier will no longer be able to associate the machine-specific state with the machine, effectively resulting in data loss. It is documented that every machine should have an unique machine-id, and strange things may happen when multiple machines with the same machine-id operate simultaneously." – Daniel T Feb 07 '24 at 03:43