7

I have a very large 'apt update' download problem (160+ MB), which occurs from time to time, especially when not updating and checking for a few days. Another update run a few hours after may not result in such a large download but only 7MB or so. Noticed in older posts about this,

that the problem was said to have been fixed, but here it is again, and i suspect the reason may be a little different from so many years back.

I have to add that I have disabled the automatic updates by setting the flags in

/etc/apt/apt.conf.d/20auto-upgrades

to 0, but I am not aware of doing any other update-related changes to the system. Further, this is not happening on all the systems running the software, one of them has normal updates all the time, under 10MB.

Here is a sample of the full terminal output:

$ apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [753 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [531 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [593 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata [295 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [382 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu bionic-updates/main DEP-11 48x48 Icons [73.8 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu bionic-updates/main DEP-11 64x64 Icons [143 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu bionic-updates i386 Contents (deb) [33.0 MB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [38.5 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/main DEP-11 48x48 Icons [17.6 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/main DEP-11 64x64 Icons [41.5 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security amd64 Contents (deb) [43.8 MB]
Get:17 http://us.archive.ubuntu.com/ubuntu bionic-updates amd64 Contents (deb) [50.3 MB]
Get:18 http://security.ubuntu.com/ubuntu bionic-security i386 Contents (deb) [28.3 MB]
Get:19 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB]
Get:20 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB]
Get:21 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 DEP-11 Metadata [253 kB]
Get:22 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe DEP-11 48x48 Icons [201 kB]
Get:23 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe DEP-11 64x64 Icons [437 kB]
Get:24 http://us.archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2,464 B]
Get:25 http://us.archive.ubuntu.com/ubuntu bionic-backports amd64 Contents (deb) [8,376 B]
Get:26 http://us.archive.ubuntu.com/ubuntu bionic-backports i386 Contents (deb) [8,376 B]
Get:27 http://us.archive.ubuntu.com/ubuntu bionic-backports/universe amd64 DEP-11 Metadata [7,920 B]
Get:28 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB]
Get:29 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB]
Get:30 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 DEP-11 Metadata [42.1 kB]
Get:31 http://security.ubuntu.com/ubuntu bionic-security/universe DEP-11 48x48 Icons [16.4 kB]
Get:32 http://security.ubuntu.com/ubuntu bionic-security/universe DEP-11 64x64 Icons [116 kB]
Get:33 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2,464 B]
Fetched 163 MB in 1min 33s (1,742 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
$

and here are only the large (MB-size) files in the above download report:

Get:12 http://us.archive.ubuntu.com/ubuntu bionic-updates i386 Contents (deb) [33.0 MB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security amd64 Contents (deb) [43.8 MB]
Get:17 http://us.archive.ubuntu.com/ubuntu bionic-updates amd64 Contents (deb) [50.3 MB]
Get:18 http://security.ubuntu.com/ubuntu bionic-security i386 Contents (deb) [28.3 MB]

And the sources list:

$ cat /etc/apt/sources.list | grep -vE "^#|^$"
deb http://us.archive.ubuntu.com/ubuntu/ bionic main restricted
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb http://us.archive.ubuntu.com/ubuntu/ bionic universe
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates universe
deb http://us.archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
deb http://us.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu bionic-security main restricted
deb http://security.ubuntu.com/ubuntu bionic-security universe
deb http://security.ubuntu.com/ubuntu bionic-security multiverse

Finally:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic
pomsky
  • 68,507
vuvu
  • 161
  • I find that updates are usually sent out almost everyday. After a few days they can add up to be quite big. Usually bigger if one or more browsers(or large programs) are updated. – crip659 Nov 08 '19 at 12:41
  • @crip659 It is not about the size of actual packages that have update available, but rather the size of the update lists (which should not be that big). – pomsky Nov 08 '19 at 12:48
  • vuvu, There are many possible solutions/workarounds suggested in those couple of posts you linked, for example switching to a different mirror. Have you tried any of those? – pomsky Nov 08 '19 at 12:52
  • @pomsky Thank you, will switch mirror and see what happens in the next days, then report back here... – vuvu Nov 08 '19 at 14:03
  • 1
    Well, i was happy for a few days with the new mirror, when this morning the 'apt update' test resulted in about 90MB download due to 2 of the big 'Contents' files mentioned above. I suppose in not too long i will see them all again. I am wondering if this is not due to having apt-file installed, which would need contents of all the debian packages... i see in the man page of apt-file that option 'update' is 'Deprecated action that just calls "apt update".' So this may be it, i will uninstall apt-file and see if it fixes the problem. – vuvu Nov 12 '19 at 12:44

2 Answers2

9

According to https://salsa.debian.org/apt-team/apt-file, "The apt command downloads the Contents files listed in its configuration.", so this is the reason for those 4 large "Contents (deb)" files being downloaded so frequently. Some 10 years ago, you had to run "apt-file update" to reload the data apt-file was relying on, now it's done automatically as a result of the system-wide "apt update" command (according to apt-file's man-page).

This explains why, after i removed the apt-file package, the download size came back to normal.

vuvu
  • 161
  • can confirm. purging apt-file fixed this issue for me as well – JoshKisb Apr 02 '20 at 20:16
  • Do you know if this mis-feature can be disabled? I would like to keep apt-file installed but only update its database when apt-file update is run a.k.a. restore the previous behavior. I don't use apt-file even weekly so I could run apt-file update if I need but there's absolutely no need to waste time to update the database every time I upgrade the system. – Mikko Rantalainen Apr 27 '21 at 14:02
  • 1
    apt update always refreshing apt-file cache can be disabled like this: sudo cp /usr/share/doc/apt-file/examples/60disable-contents-fetching.conf /etc/apt/apt.conf.d/60-apt-file-disable-autoload.conf. For details, see man 1 apt-file and less /usr/share/doc/apt-file/README*. – Mikko Rantalainen Apr 28 '21 at 14:50
  • @MikkoRantalainen : thank you for the hint. Sounds like the right thing to do indeed. Will try it sometime these days and upgrade my answer to reflect it. – vuvu Apr 29 '21 at 02:47
  • @MikkoRantalainen : installed back apt-file and copied 60disable-contents-fetching.conf as you said above [and as also indicated in README file you mentioned] to apt.conf.d, but today i got another 444MB download, despite these precautions. So, it's not working. Now i am trying, as a last resort, a brute-force solution: the only reason apt downloads the Contents files seems to be the 50apt-file.conf file, which apt-file install places in apt.conf.d, so i moved this file elsewhere and i expect NOT to see any more large updates until i need an update of apt-file itself and i move it back... – vuvu May 19 '21 at 12:53
  • Which Ubuntu version you're using? I have 18.04 LTS. – Mikko Rantalainen May 20 '21 at 21:56
  • Ubuntu 18.04.5 LTS – vuvu May 21 '21 at 22:46
0

I had to purge the apt-file package instead of just removing it:

sudo apt purge -y apt-file

purge is like a deep remove. Quoting man apt-get:

purge is identical to remove except that packages are removed and purged (any configuration files are deleted too).

In my case culprit was file /etc/apt/apt.conf.d/50apt-file.conf, and a simple remove was not uninstalling it

Note: for those who don't know the apt-file package, it is a tool to query the apt database (search which package provides a given file, etc...)

Elouan Keryell-Even
  • 160