25

I have configured unattended-upgrades to install security packages and notify by mail when it does so.

I have noticed that the installation happens at very random times. I know that the latest versions added a random delay up to 30 minutes starting from the cron.daily execution time.

However the delays I am experiencing are much bigger than that. I see unattended-upgrades executing at 9am, 3pm, 12am... The logs show the same, so it is not just the email delivery that takes longer.

The unattended upgrades task is the first one in cron.daily, meaning that there is no previous task with huge execution times.

Anybody experienced a similar thing?

daniel f.
  • 846
  • The random behavior is deliberate - to smooth out demand instead of millions of systems hammering a few mirrors at the same times each day. Ordinary desktop users should not notice the behavior at all. Some enterprise users way wish to change the behavior to something a bit more predictable, and are certainly welcome to do so. – user535733 Feb 05 '19 at 13:50
  • Yes, the reason behind this choice is clear. It is just that this behavior is unacceptable for production systems. At the time i asked this question, this behavior (and the fix) was not documented anywhere – daniel f. Feb 05 '19 at 14:06

5 Answers5

35

After debugging this I found the solution.

The root cause of this issue resides in the fact that under Ubuntu 16.04 and newer, unattended-upgrades uses systemd - not cron - to schedule the updates with a huge randomized delay:

/lib/systemd/system/apt-daily.timer is configured with

OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h

This means that it will run twice a day, at 6:00 and 18:00, with a random delay of up to 12 hours. As this is not always acceptable for production environments, I had to override these settings.

In order to keep the package config files untouched, I defined my override in /etc/systemd/system/apt-daily.timer.d/override.conf (Update: please read the edit at the bottom of this answer for further information on filename and location, as it seems to be slightly subject to change).

There I set

[Timer]
OnCalendar=
OnCalendar=06:00
RandomizedDelaySec=1h

to have unattended-upgrades run at 6:00 plus a random delay of up to an hour.

Then I simply restarted the timer with systemctl restart apt-daily.timer (eventually need to reload the daemon).

Unattended-updates now runs at predictable times again!

Edit: It would seem as if for Ubuntu 18.04 things changed a bit. The override should now be stored in /etc/systemd/system/apt-daily-upgrade.timer.d/override.conf and look like this:

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=1h

@PerlDuck has mentioned a way of creating an override file with the right name and location in a comment below. Instead of manually creating a file, please consider running sudo systemctl edit apt-daily.timer

daniel f.
  • 846
  • 1
    Why do you clear OnCalendar first? – jarno Dec 14 '16 at 10:15
  • 3
    Because otherwise it would only ADD a new timer at 6am, leaving the existing one as well. Since I want unattended-upgrades to ONLY run at 6, I need to clear the schedule first. – daniel f. Dec 14 '16 at 10:19
  • I added "OnBootSec=5min" to enable running after boot as well, but it did not work. (also added OnUnitActiveSec=12h so that it is not run too frequently.) – jarno Dec 14 '16 at 15:09
  • systemd, or rather, the systemd mindset, strikes again. I may have to rethink upgrading to Xenial Xerus in production after this gem bit me. – Joe Jan 11 '17 at 11:30
  • 2
    @daniel f. I have an apt-daily.timer file in /lib/systemd/system/ as well as one in /etc/systemd/system/timers.target.wants/ However I don't have any in /etc/systemd/system/ itself like you. Would you know if I should be creating the apt-daily.timer.d directory and the override.conf under /lib/systemd/system instead? All advice gratefully recd with thx. – Purvez Apr 27 '17 at 10:24
  • When running systemctl restart apt-daily.timer I get Warning: The unit file, source configuration file or drop-ins of apt-daily.timer changed on disk. Run 'systemctl daemon-reload' to reload units., so it seems you have to run that instead. – simlev May 14 '18 at 09:16
  • Why in first example you used "OnCalendar=" and in the second one you didn't? Also why in first example you removed the "--*" and in the second one you didn't? – red-o-alf Mar 14 '20 at 13:06
  • Under 18.04LTS, I just ran sudo systemctl edit apt-daily.timer. It created /etc/systemd/system/apt-daily.timer.d/override.conf. I will check if it works properly. – sancho.s ReinstateMonicaCellio Apr 13 '20 at 11:02
12

The best way to update the unattended upgrade time, as I compiled it from various sources and tested on our system, is to exclusively use the systemctl commands and avoid trying to find the proper files to edit.

The only thing you have to know for sure is the service name, which in our case, is apt-daily-upgrade (if unsure, search for it via $ systemctl | grep apt). When a systemd service has a timer defined, it is referenced as #{service_name}.timer, thus it’s apt-daily-upgrade.timer for us.

As the system configuration should not be edited, we’ll have to override the default timer config in systemd. For this you need to copy-edit some parts of the original config, so let’s show it first:

$ systemctl cat apt-daily-upgrade.timer 
# /lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer] OnCalendar=--* 6:00 RandomizedDelaySec=12h Persistent=true

[Install] WantedBy=timers.target

We’ll need to update the OnCalendar and RandomizedDelaySec values in the [Timer] section. Let’s create the override config file via the following command:

$ systemctl edit apt-daily-upgrade.timer

This should open an editor with a blank file and we need to put there the amended [Timer] section, at least:

[Timer]
# Reset the system calendar config first
OnCalendar=
# Set a new calendar timer with a 60 minute threshold
OnCalendar=*-*-* 21:00
RandomizedDelaySec=60m

As you can see, we’ve updated the OnCalendar value to trigger the automatic updates in the evenings, instead of mornings. The blank OnCalendar line above it must be present as this config value is additive, i.e. it may be specified more than once and only setting it to a blank value resets all previous OnCalendar values (the ones from the system config).

Once we save the file, we can verify that systemd knows about it (there’s no need to run systemctl daemon-reload, the edit command does that for us upon leaving the editor) by running systemctl again as above:

$ systemctl cat apt-daily-upgrade.timer

/lib/systemd/system/apt-daily-upgrade.timer

[Unit] Description=Daily apt upgrade and clean activities After=apt-daily.timer

[Timer] OnCalendar=--* 6:00 RandomizedDelaySec=12h Persistent=true

[Install] WantedBy=timers.target

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf

[Timer]

Reset the system calendar config first

OnCalendar= OnCalendar=--* 21:00 RandomizedDelaySec=60m

Now it shows two configurations, with our custom one overriding the default one. Good!

The final check that it all works as expected can be done via the list-timers command of systemctl:

$ systemctl list-timers
NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES                     
...
Thu 2020-08-06 21:51:36 UTC 12h left      Thu 2020-08-06 07:10:20 UTC 2h 20min ago apt-daily-upgrade.timer      apt-daily-upgrade.service               
...             

Find the proper line in the output and look at the NEXT column - the value in there should reflect the time of your newly configured unattended upgrade.

LSerni
  • 433
  • 6
  • 6
Matouš Borák
  • 221
  • 2
  • 5
10

The official debian documentation on https://wiki.debian.org/UnattendedUpgrades currently has a mistake in it that is misleading a lot of people. It claims that you can override the upgrade time by creating a file called

/etc/systemd/system/apt-daily-upgrade.d/override.conf

However the correct path is

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
PerlDuck
  • 13,335
  • 3
    Nice finding. IMHO the safest thing is to use sudo systemctl edit apt-daily.timer. This will open an editor with the correct drop-in file. – PerlDuck Feb 05 '19 at 10:39
  • 2
    Thank you PerlDuck, I edited the Debian Wiki page with your suggestion – Rolf Wojtech Feb 05 '19 at 12:45
  • Upvoted this answer for being useful – and for updating the Debian wiki page! – Anthony Geoghegan Feb 27 '20 at 21:16
  • @PerlDuck - Under 18.04LTS, I just ran sudo systemctl edit apt-daily.timer. It created /etc/systemd/system/apt-daily.timer.d/override.conf... Why would it be so? I will check if it works properly. – sancho.s ReinstateMonicaCellio Apr 13 '20 at 11:04
  • @sancho.sReinstateMonicaCellio Maybe you upgraded your 18.04 from a previous version instead of doing a fresh install. I reckon they didn't change the location of the unit files during an update but only use the new location (/lib/… instead of /etc/…) in a fresh install. Just a guess, though. Still: if systemctl edit … thinks /etc/… is the correct path, then you can trust it. I wouldn't bother. You can also try systemctl cat … to see the current path (and content) of that unit. (BTW: I'm not the author of this answer, I just fixed some formatting.) – PerlDuck Apr 14 '20 at 11:22
  • @PerlDuck - Great. I know. – sancho.s ReinstateMonicaCellio Apr 14 '20 at 13:24
8

I tried Daniel's solution but the upgrade still ran at incorrect times. Turned out there are two systemd overrides needed:

Used for downloads

/lib/systemd/system/apt-daily.timer

...that is overridden with:

/etc/systemd/system/apt-daily.timer.d/override.conf

Used for upgrading

/lib/systemd/system/apt-daily-upgrade.timer

...that is overridden with:

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
0

The apt-daily.timer is set to run twice a day, and will not interrupt a production environment, as it simply updates and downloads the packages that need to be updated. You should only need to override apt-daily-upgrade.timer to a time that would be less likely to impact your production environment as this is when the updates are actually installed. Further, there is a setting to reboot for things that require it in 50unattended-upgrades that you should consider setting so they actually get applied and that also has a timer. Also, as of 22.04 the upgrade timer has been changed to default 6am + 1hr, so most will not need to change this.