Making mirror://mirrors.ubuntu.com highly available

Question

We're a distributed team, which is why our VMs use the Ubuntu mirror:// setup. Our /etc/apt/sources.list looks like the following:

deb mirror://mirrors.ubuntu.com/mirrors.txt lucid main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt lucid-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt lucid-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt lucid-security main restricted universe multiverse

This by itself is pretty amazing and very helpful for people who work in different locations — less local customization, etc. necessary. Fail-over in theory.

In day-to-day, this setup fails too often. I want to say 2-3 times this week.

Right now mirrors.ubuntu.com returns ftp.uni-bayreuth.de as my closet mirror. Unfortunately, it seems to be down.

This has been going on for a few hours and the mirror is hosted by volunteers at a university and today being a Friday, my hopes are low that this is getting corrected soon.

All talk, my questions are:

• Does anyone use this?
• How do you work around downtime? (my quick-fix is a shell script)
• How can I help to improve this situation?

Answer 1

Personally I think that the best way to select the best Ubuntu repositories mirrors is to use the GUI method:

Now, to improve the situation described in the question, you need to set some rules somehow. These rules must to act on mirrors.ubuntu.com. I can suggest some rules as follow:

• make a list of best/preferred mirrors; there are a lot of mirrors as you can see here, here or here
• if you found a good mirror, add it to the list
• if a mirror was down or broken sometimes, it means that is not a good mirror and you should remove it from the list
• you can use netselect, apt-spy or apt-fast
• and others, depending on your requirements.

Next, to see how you can work around, I can give you a method described step by step with three bash scripts examples. First script use the mirrors from the country you are in at the moment instead of mirrors.ubuntu.com/mirrors.txt (for each country there is a text file with mirrors asociated; see http://mirrors.ubuntu.com/):

• In a terminal run mkdir -p bin - this command will make a bin directory in your home folder if you don't already have it.
• After run gedit ~/bin/change_sources.sh - this will create the new file change_sources.sh in gedit.
• Copy and paste one of the next scripts in the new created file:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

ip=$(curl -s 'http://ipecho.net/plain')
country=$(curl -s 'http://geoiplookup.net/geoapi.php?output=countrycode' \
    | awk '{ print toupper($2) }')
release=$(lsb_release -sc)

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

line=$(head -n 1 $file)
new_line="## Ubuntu Repos for $ip"

if [ "$line" == "$new_line" ] ; then
    exit 0
fi

cp -f $file $old_file

printf "$new_line
deb mirror://mirrors.ubuntu.com/$country.txt $release main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-security main restricted universe multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

or, something similar to what can be found at http://repogen.simplylinux.ch/:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

ip=$(curl -s 'http://ipecho.net/plain')
country=$(curl -s 'http://geoiplookup.net/geoapi.php?output=countrycode' \
    | awk '{ print tolower($2) }')
release=$(lsb_release -sc)

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

line=$(head -n 1 $file)
new_line="## Ubuntu Main Repos for $ip"

if [ "$line" == "$new_line" ] ; then
    exit 0
fi

cp -f $file $old_file

printf "$new_line
deb http://$country.archive.ubuntu.com/ubuntu/ $release main restricted universe  multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release main restricted universe multiverse

## Ubuntu Update Repos for $ip
deb http://$country.archive.ubuntu.com/ubuntu/ $release-security main restricted universe multiverse
deb http://$country.archive.ubuntu.com/ubuntu/ $release-updates main restricted universe multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release-security main restricted universe multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release-updates main restricted universe multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

or, a script using netselect (download from here, install instructions here) as izx explained verry nice in this answer:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

url=$(netselect \
    `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors \
        | grep -P -B8 "statusUP|statusSIX" \
        | grep -o -P "(f|ht)tp.*\"" \
        | tr '"\n' '  '` \
    | awk '{print $2}')
release=$(lsb_release -sc)

if [ "$url" == "" ] ; then
    exit 0
fi

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

cp -f $file $old_file

printf "## Ubuntu Best Repos
deb http://extras.ubuntu.com/ubuntu $release main
deb-src http://extras.ubuntu.com/ubuntu $release main
deb $url $release main universe restricted multiverse
deb http://security.ubuntu.com/ubuntu/ $release-security restricted universe main multiverse
deb $url $release-updates restricted universe main multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

• Save the file and close it.
• Go back into terminal and run: chmod +x ~/bin/change_sources.sh - to grant execute access for the script.
• Just for test, to run your new script, type in terminal ~/bin/change_sources.sh. It will give you an error, because you don't have the right to edit /etc/apt/sources.list. So, use sudo ~/bin/change_sources.sh
• Edit the root user's crontab file using sudo crontab -e command and add the following line:

@hourly /home/$USER/bin/change_sources.sh  
#change $USER with your user name

• I have set the cron job for every hour, but you can change as you wish or as you think is better. See http://en.wikipedia.org/wiki/Cron in this sense.
• Save the file and check the new crontab entry with sudo crontab -l.

NOTE: To revert the changes made by this script, delete the cron job and follow the indications from the picture above or use next command in terminal:

cp -f /etc/apt/sources.list.bak /etc/apt/sources.list

From now, the file will be dynamically changed after it finds a change of the IP address.

It may not be the best solution, but, in my opinion, a good solution can be given in this way like in the scripts above.

Answer 2

I appreciate all the input on this question, but since no one came up with a simple solution which fit our circumstances, I decided to fix the problem myself.

I created a tool (specifically for Ubuntu) which I call apt-spy2.

The primary objective of this tool is to find a working mirror fast. Working is defined by that the mirror server is available and (hopefully :) up to date.

I make no assumptions about if the selected server is necessarily the closest and the fastest. I'm not doing any pings or GEO DNS tricks — but so far this works when something breaks.

How it works — in a nutshell:

1. I use http://mirrors.ubuntu.com or launchpad's list of mirrors to retrieve servers.
2. I do a simple check on each (for HTTP Response Status Code).
3. LBNL, I update /etc/apt/sources.list.

Please note: This assumes that people play nice and put additional mirrors (e.g. 3rd party repositories into /etc/apt/sources.list.d. But I guess that means there's room for improvement.

You can obtain this tool like so:

$ [sudo] gem install apt-spy2

The cli comes with list, check, fix and help (with extended information on how to use it).

I tried to document as much as possible in the project's README.

The current version is a very conservative 0.5.0.

The code is open source and the license is liberal. And I take all contributions.

Answer 3

There was this command at Debian 6.0.4 :

apt-spy

This did the job to find the next closest available server automatic and to generate a new sources.list

In Ubuntu this command seems not to exist ?

It still exists in Debian 7.0 wheezy :

https://launchpad.net/debian/wheezy/+source/apt-spy/+copyright

You can download your *.deb package here :

http://packages.debian.org/sid/apt-spy

... still searching for the sources ...

You obviously need Debian-7.0-Installation for to get the source-code after editing sources-list with entry :

deb-src http://http.debian.net/debian wheezy main

Then after sudo apt-get update you would simply suck the code with :

sudo apt-get source apt-spy

Answer 4

Maybe not using the solution you like, but setting up a local cache for the .deb files should be ideal when you download several MB of package (mostly of them I bet are the same) and since you are downloading from the same server, you might like to sacrifice a PC to make the cache and make the server less busy.

Procedure

You should be able to set apt-cacher in a machine that have lots of space in the Hard Drive (you would need it). You can configure this using the Apt-cacher server guide, but lets do a quick resume.

1. Install apt-cacher

   sudo apt-get install apt-cacher apache2

2. Edit /etc/default/apt-cacher and set the autostart value to 1, so your cache starts when the machine power up.
3. Restart apache sudo /etc/init.d/apache2 restart
4. Test your cache using http://ip-of.your.cache:3142/apt-cacher.
5. You should allow host to use the cache, so edit /etc/apt-cacher/apt-cacher.conf and look for the line allowed_hosts. You can set subnets like 192.168.0.0/24 or just * for all hosts
6. Look for a fast and reliable server around. You could use apt-spy for this (this command could take hours since it also test the speed of each server).

7. In your clients you have two options:

   • set the sources.list as follow:

   deb http://ip-of.your.cache:3142/the.server.you.like/ubuntu/ lucid main restricted universe multiverse

   Change the ip-of.your.cache for the ip of your cache and the.server.you.like for the host name of the server you will be using. You can use several lines.

   • Or set apt to use a proxy: edit /etc/apt/apt.conf.d/01proxy and add the line:

   Acquire::http::Proxy "http://ip-of.your.cache:3142";

Answer 5

You can try setting up your own Ubuntu mirror. That way updates will be local.

apt-mirror needs a high speed internet connection and a lot of disk space

Setting up your own Ubuntu mirror is very easy by using apt-mirror.

1) go into the apt-mirror repository. Back up your sources.list file like so:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.old

2) Edit the sources.list file like so:

gksudo gedit /etc/apt/sources.list

3) And insert the following on a new line:

deb http://apt-mirror.sourceforge.net/ apt-mirror

4) Let’s update the package index

sudo apt-get update

5) And install apt-mirror

sudo apt-get install apt-mirror

6) Apt-mirror is now set up so let’s tell it what servers to mirror:

sudo cp /etc/apt/mirror.list /etc/apt/mirror.list.old
gksudo gedit /etc/apt/mirror.list

The default configuration will put your downloaded files in the /var/spool/apt-mirror folder. Here’s how the mirror.list file looks like:

############# config ##################
#
set base_path /media/silo/repo
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
set nthreads 20
set tilde 0
#
############# end config ##############
deb http://archive.ubuntu.com/ubuntu lucid main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid -updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid -backports main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid -security main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu

this only mirrors binary packages but if you want source packages, you should insert the appropriate lines. They usually take the form:

deb-src http://gh.archive.ubuntu.com/ubuntu/ lucid main restricted

7) Now, run apt-mirror. This is going to take quite a while. Don’t worry if you cannot download all the files in one go, apt-mirror can resume downloads (kill it with the Ctrl+C key combination and re-run it when you want to continue). Okay, run apt-mirror like so:

sudo apt-mirror /etc/apt/mirror.list

Also check out How To Create A Local Debian/Ubuntu Mirror With apt-mirror

^{Source:Use Apt-mirror to Create Your Own Ubuntu Mirror}

Answer 6

I've written a Python script called apt-select to serve as a CLI alternative to the GUI method.

The script queries each mirror via TCP, prints to stdout mirrors with the lowest latency, and generates a new sources.list file. There are also multiple options that allow you to choose from a list of ranked mirrors, and retrieve information such as when the mirror was last updated and its bandwidth capacity.