0

The context is a bug in which Ubuntu are updating package list timestamps every 30 minutes on their archive mirrors, which forces [apt-get|aptitude] update commands to download package lists though they haven't changed:

The proposed workaround is to run touch over the affected index files prior to running apt-get update. This make their timestamps current, and avoids having to re-download them. This speeds processing by up to several minutes (2-3 seconds vs. 2-6 minutes in various tests). This despite running a local apt-cache-ng proxy.

We're running into this testing (mostly) and deploying (somewhat) Ubuntu-based images for our AWS infrastructure in which apt-get update runs a couple of times. 3-6 minutes vs. 3-20 seconds makes a big difference in our iteration time.

It seems odd to me that apt doesn't rely on filesize or hashes to determine whether or not files have changed. Timestamps are very trivially modified, and seem awfully brittle to base this behavior on.

Community
  • 1
Dr. Edward Morbius
  • 101
  • 3
  • You shouldn't need to run apt-get update every thirty seconds, so I don –  Jun 28 '12 at 21:43
  • 1
    The suggestion to 'touch' things isn't a proposed 'fix'. It is a hacky work-around. A much better work-around is suggested, use a HTTP proxy. – Zoredache Jun 28 '12 at 22:05
  • ionFish: I've just edited my question to address that. We're testing AWS deployments and the issue comes up there. Zoredache: Noted, edited in question. And, actually, we're using apt-cacher-ng. I've confirmed watching both the client and the cache that we're hitting the origin. – Dr. Edward Morbius Jun 28 '12 at 22:21

1 Answers1

1

When I first tested the touch hack, I was surprised that apt-get used the local file mod time to inform the http server what to download--but that's obviously the case (or the hack wouldn't work).

The downloaded Release file contains a reliable date and these files are crytographically signed, and are therefore quite trustworthy, but the hashes aren't used to determine what package files to download.

With the timestamp approach now in use for package files, there's nothing special needed from the mirror site. We owe the Debian project for the design. It's helpful to mirror owners to keep Ubuntu identical to Debian and keep things vanilla and therefore easy to manage.

This design normally works fine, as far as I know. This is just a bug which should be fixed as far as I'm concerned, but the bug is in Canonical private code so outsiders like me can't help patch it. I hope they do so soon as we are abusing the hospitality of those running Ubuntu mirrors, in addition to causing unnecessary expense for some Ubuntu users who must pay for bandwidth by the gigabyte.

There is an http protocol mechanism for using an etag to represent a served item, but it's not used for apt-get. Such a tag could contain the hash from the Release file.

It's the large number of packages included in Ubuntu releases which make this bug so significant.

From the signed (and therefore checksummed) Release, there are hashes for these files, and in these files hashes for versions of packages and from there checksums for files. Since the package files contain the version numbers, and since they don't change, the bug doesn't cause redundant package upgrades.

John S Gruber
  • 13,336
  • 1
    How do you suggest that the ETags would get updated in a way that applies to all web servers? Keep in mind, that not everybody is running Apache, you need a extremely basic web server to support APT repositories. – Zoredache Jun 29 '12 at 00:01
  • @Zordache -- I'm not proposing it, just trying to say the protocol mechanism exists in case the questioner had that in the back of his mind. – John S Gruber Jun 29 '12 at 00:06
  • @Zordache -- may I suggest that you post an answer to the apt-get list question above containing your proxy and rsync approaches? I understand that Debian was or is considering a file delta approach to file size issues, but that Ubuntu wasn't as of last month. – John S Gruber Jun 29 '12 at 00:09