8
ls -Ssh /var/lib/apt/lists/partial | head

total 88G
 46G security.ubuntu.com_ubuntu_dists_vivid-security_main_binary-amd64_Packages.bz2
 42G us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2

I tried bunzipping and got trailing garbage:

bunzip2: us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2: trailing garbage after EOF ignored

It looks like the decompressed file is only 88K:

 du -hs us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages
88K

Doing a hex dump of the end of the file appears to reveal a directory listing being appended:

tail -c 200 security.ubuntu.com_ubuntu_dists_vivid-security_main_binary-amd64_Packages.bz2 | hexdump -C
00000000  30 38 64 66 37 20 20 20  20 20 20 20 20 20 20 20  |08df7           |
00000010  20 20 34 38 31 39 20 75  6e 69 76 65 72 73 65 2f  |  4819 universe/|
00000020  69 31 38 6e 2f 54 72 61  6e 73 6c 61 74 69 6f 6e  |i18n/Translation|
00000030  2d 65 6e 2e 62 7a 32 0a  20 31 30 66 64 62 31 61  |-en.bz2. 10fdb1a|
00000040  37 34 61 62 34 34 33 36  64 39 37 62 33 33 66 65  |74ab4436d97b33fe|
00000050  62 30 38 38 36 66 35 63  38 61 63 63 64 36 34 34  |b0886f5c8accd644|
00000060  33 62 36 34 36 37 61 30  64 63 39 61 63 39 64 31  |3b6467a0dc9ac9d1|
00000070  66 64 33 39 61 36 31 63  62 20 20 20 20 20 20 20  |fd39a61cb       |
00000080  20 20 20 20 20 20 31 35  38 31 20 75 6e 69 76 65  |      1581 unive|
00000090  72 73 65 2f 73 6f 75 72  63 65 2f 53 6f 75 72 63  |rse/source/Sourc|
000000a0  65 73 2e 62 7a 32 0a 20  61 64 66 35 31 32 30 66  |es.bz2. adf5120f|
000000b0  65 63 61 37 38 66 34 65  37 36 66 34 31 36 66 38  |eca78f4e76f416f8|
000000c0  31 30 32 31 36 62 33 30                           |10216b30|

Piping the tail though strings and greping for a file shows the same file listed over and over:

 tail -c 500K security.ubuntu.com_ubuntu_dists_vivid-security_main_binary-amd64_Packages.bz2 | strings  | grep -i multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2
 64a543afbb5f4bf728636bdcbbe7a2ed0804adc2               14 multiverse/i18n/Translation-en.bz2
 d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058               14 multiverse/i18n/Translation-en.bz2

Has anyone run into this before?

Wilf
  • 30,194
  • 17
  • 108
  • 164
user402751
  • 81
  • 1
  • 4
  • I found a workaround to the issue: rm -rf /var/lib/apt/lists/partial/* and then re-run apt-get update. – user402751 Apr 28 '15 at 16:18
  • I ran into this again, after doing fresh install from DVD, I originally had upgraded from 14.10. {{392G us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2}}} The signature is a little different, there are repeated http errors. {{tail -c 500K us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2 | strings | grep "HTTP/1.1 416 Requested Range Not Satisfiable" HTTP/1.1 416 Requested Range Not Satisfiable HTTP/1.1 416 Requested Range Not Satisfiable HTTP/1.1 416 Requested Range Not Satisfiable HTTP/1.1 416 Requested Range Not Satisfiable }} – user402751 May 06 '15 at 14:01
  • Hmm, I seem to have this problem as well. Very annoying. I come to work in the morning, and all sorts of things have crashed because the disk is full. Have you by chance found a solution? – Frode May 19 '15 at 09:17
  • Solution, no. I've just been using the workaround of removing all the files in /var/lib/apt/lists/partial/* prior to "apt-get update" – user402751 May 21 '15 at 03:28
  • It's a bug. http://askubuntu.com/a/633235/43246 – aleung Jul 31 '15 at 03:51

1 Answers1

6

Sounds a bit like a proxy problem. I once had a problem with maven that the proxy replaced a jar by a html page containing a virus warning. The html got saved as .jar but because it was broken nothing worked anymore.

Anyway: it's safe to delete everything from the directory /var/lib/apt/lists/partial/ (and also from /var/cache/apt/archives/partial/).

For troubleshooting, I recommend the following steps:

  • Try another mirror, eg. http://de.archive.ubuntu.com/ubuntu/ instead of http://ch.archive.ubuntu.com/ubuntu/ or even the main archive http://archive.ubuntu.com/ubuntu/
  • Try another protocol: ftp:// instead of http:// - the rest of the path remains for default mirrors
  • Try to analyze the transfer using wireshark (use the context menu option 'Follow TCP stream'). Interesting are both HTTP headers and content. If you get answers with HTTP status code 200 (which means ok) but human-readable content, something is seriously wrong on your proxy. A regular bzip2 file shouldn't contain any human-readable text.
  • Use wget or curl for checking what you get from the server when downloading without apt-get.
Daniel Alder
  • 2,426
  • Interesting, I am in fact behind a proxy. @user402751 , are you as well? – Frode May 24 '15 at 11:47
  • Yes, I'm behind a proxy as well which probably explains why not everyone runs into this issue. That being said there is a pretty bad bug here as it just writes the disk until full. – user402751 May 24 '15 at 15:56
  • 2
    Proxy support is not per se broken, but problably some servers don't behave correctly. Did you ever use Wireshark for analyzing what's transported while downloading (Option 'Follow TCP stream)? Interesting are both HTTP headers and content. Normally, a bzip2 file shouldn't contain any usable output when piping through the string command. Just to make sure: are you using http or ftp servers? – Daniel Alder May 25 '15 at 11:04
  • I'm curious? Did this answer help anyone? Did user402751 or @Frode try the steps? – Daniel Alder Jun 01 '15 at 11:58
  • @DanielAlder I did not pursue the problem further (yet). It's at least helpful to know that it's probably a proxy problem. – Frode Jun 02 '15 at 15:18