KVM Bridged Network Not Working

Question

I just installed KVM on my Ubuntu Server according to this guide : https://help.ubuntu.com/community/KVM/Installation

Then prepared a bridged network as shown in here : https://help.ubuntu.com/community/KVM/Networking

Then, I created a virtual machine with virt-manager. I tried several times but the guest fails to connect to the network! Any help?

ifconfig :

      br0       Link encap:Ethernet  HWaddr d0:27:88:b0:e4:38  
                inet addr:192.168.20.100  Bcast:192.168.20.255  Mask:255.255.255.0
                inet6 addr: fe80::d227:88ff:feb0:e438/64 Scope:Link
                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                RX packets:62 errors:0 dropped:0 overruns:0 frame:0
                TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:0 
                RX bytes:10493 (10.4 KB)  TX bytes:8433 (8.4 KB)

      eth0      Link encap:Ethernet  HWaddr d0:27:88:b0:e4:38  
                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                RX packets:62 errors:0 dropped:0 overruns:0 frame:0
                TX packets:63 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:1000 
                RX bytes:11361 (11.3 KB)  TX bytes:8479 (8.4 KB)
                Interrupt:41 

      lo        Link encap:Local Loopback  
                inet addr:127.0.0.1  Mask:255.0.0.0
                inet6 addr: ::1/128 Scope:Host
                UP LOOPBACK RUNNING  MTU:16436  Metric:1
                RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:0 
                RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

      virbr0    Link encap:Ethernet  HWaddr 5a:8c:57:95:af:3b  
                inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
                UP BROADCAST MULTICAST  MTU:1500  Metric:1
                RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:0 
                RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

brctl show :

 bridge name    bridge id      STP enabled    interfaces
 br0       8000.d02788b0e438   no        eth0
 virbr0         8000.000000000000   yes  

brctl showmacs br0 :

 port no   mac addr       is local? ageing timer
   1  5c:d9:98:67:b6:28   no          48.33
   1  d0:27:88:b0:e4:38   yes          0.00
   1  e0:2a:82:f9:6c:09   no           0.00

ip route :

 default via 192.168.20.1 dev br0  metric 100 
 192.168.20.0/24 dev br0  proto kernel  scope link  src 192.168.20.100 
 192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1

*In the guest * I was unable to copy paste the info from the guest because can't ssh to it. It didn't get any ip from DHCP. Won't work even after setting it up manually.

Answer 1

Preliminaries

This following worked for me for Ubuntu 12.04. You should disable your computer's firewall as you test this so it won't interfere.

The /etc/default/qemu-kvm file should be as originally installed.

You will need to have bridge-utils qemu-kvm and libvirt-bin installed. Any users using virtual machines should be added to the libvirtd group.

There no longer seems to be any need to add CAP_NET_ADMIN capability.

Network Setup

The default network mode is the User mode, also called SLIRP. It uses a predefined virbr0 bridge which is NAT routed to the guest computer. The NAT routing uses the kernel's ip_forwarding feature and iptables. Bridge mode uses a virtual bridge in the guest to which the (unnumbered) Ethernet interface connects, and on which both the host and the guest have their network interfaces.

The following diagrams may make the differences clearer:

You can see how the default User network is defined with:

virsh net-dumpxml default

I can set up the bridged mode with the following approaches:

In /etc/network/interfaces (from the bridging part of the post you mention in your question):

auto lo
iface lo inet loopback
#auto eth0
#iface eth0 inet dhcp
auto eth0
iface eth0 inet manual
auto br0
iface br0 inet dhcp
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

Reboot; and make sure that wireless networking isn't active. Check the default IP route with ip route. It must be using the br0 interface.

N.B. If your Ethernet isn't hooked up when this change is made you need to have your Ethernet cable plugged in and getting a carrier or the boot will hang for two minutes and you won't have network capability That's because the eth0 interface, by being in this file, must come up before the boot can proceed normally.

N.B. Generally you can't use a wireless network instead of eth0 because of their inability to use multiple MAC addresses (I infer that they need a second one for the bridge).

As an alternative you can disable the use of Ethernet and make sure that it does not have an IP address, and that there isn't a default route set up with ip route. Then:

 sudo ifconfig eth0 0.0.0.0 up
 sudo brctl addbr br0
 sudo brctl addif br0 eth0
 sudo ifconfig br0 up
 sudo dhclient br0 &

You could also supply a static IP address here, as well as defining the default route and DNS address. For this example dhclient does this.

Here's my route table:

$ip route list
default via 192.168.1.1 dev br0  metric 100 
169.254.0.0/16 dev br0  scope link  metric 1000 
192.168.1.0/24 dev br0  proto kernel  scope link  src 192.168.1.45 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1

Using kvm

I can then boot a bridged kvm machine with:

 $ sudo kvm -name Quantal -m 1024 -hda foo.qcow2 --soundhw ac97 -device virtio-net,netdev=tunnel -netdev tap,id=tunnel,ifname=vnet0

The -netdev tap parameter makes sudo a requirement. As the VM is started qemu-kvm runs the following commands:

ifconfig vnet0 0.0.0.0 up
brctl addif brctl addif br0 vnet0

This is done by /etc/qemu-ifup

The VM's vnet0 interface is added to the br0 bridge because the default route above uses that bridge interface. If it weren't there the tap interface instead would be added to the virbr0 interface. Since that's not connected to the Internet, NAT would be used to connect the guest to the host and the Internet, in my experiments. You can direct the vnet0 to a particular bridge in /etc/default/qemu-kvm. Using virt-manager below you can explicitly direct which bridge to connect to.

Because of the above commands issued by qemu-kvm, and the -netdev tap,id=tunnel,ifname=vnet0 parameter, the vm virtual machine is connected to the vnet0 tunnel, and the tunnel is connected to the br0 bridge.

I can now directly ssh into this guest VM from another computer on my network.

My host ifconfig (note the vnet0 interface that appears on my network when the VM is running):

$ifconfig
br0       Link encap:Ethernet  HWaddr 00:1e:33:88:07:e5  
          inet addr:192.168.1.45  Bcast:255.255.255.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:33ff:fe88:7e5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6526 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7543 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2712940 (2.7 MB)  TX bytes:1071835 (1.0 MB)

eth0      Link encap:Ethernet  HWaddr 00:1e:33:88:07:e5  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7181 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7740 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2974585 (2.9 MB)  TX bytes:1096580 (1.0 MB)
          Interrupt:43 Base address:0x6000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:664 (664.0 B)  TX bytes:664 (664.0 B)

vnet0      Link encap:Ethernet  HWaddr ca:0c:73:c3:bc:45  
          inet6 addr: fe80::c80c:73ff:fec3:bc45/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:226 errors:0 dropped:0 overruns:0 frame:0
          TX packets:429 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:26919 (26.9 KB)  TX bytes:58929 (58.9 KB)

virbr0    Link encap:Ethernet  HWaddr d6:18:22:db:ff:93  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

My bridge configuration while running the VM:

$brctl show
bridge name bridge id       STP enabled interfaces
br0             8000.001e338807e5       no              eth0
                                                        vnet0
virbr0          8000.000000000000       yes

Note that both the virtual machine's vnet0 interface and the eth0 interface are connected to the br0 bridge.

And the MAC's on the br0 interface:

$brctl showmacs br0
port no mac addr        is local?   ageing timer
  1 00:05:5d:cf:64:61   no         2.54
  1 00:19:d2:42:5d:3f   no        36.76
  1 00:19:df:da:af:7c   no         2.86
  1 00:1e:33:88:07:e5   yes        0.00
  1 00:60:0f:e4:17:d6   no         0.79
  2 52:54:00:12:34:56   no         0.80
  1 58:6d:8f:17:5b:c0   no         5.91
  1 c8:aa:21:be:8d:16   no       167.69
  2 ca:0c:73:c3:bc:45   yes        0.00

Note that the br0 interface connects my host computer to the same bridge being used by the guest.

You can check that you are bridged rather than NAT routed to your own network by using traceroute 8.8.8.8. If the first node is your network's router rather than the guest's ip address your network should be working correctly.

See this documentation.

virt-manager

Be sure that you have installed virt-manager and hal. The hal package is a suggested dependency for virt-manager and is used to determine the network configuration of your system when creating or editing guests.

While having the br0 bridge defined as above I created a virtual machine with virt-manager as follows:

I was able to go directly to the rest of my home network and to the Internet from this guest. I was also able to ssh into it from the other (non-host, non-guest) Ubuntu computer on my home network.

Here's the very long kvm command run by virt-manager (for comparison with EApubs or anyone else having trouble with this):

/usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name precise -uuid f057a729-eda6-4b85-84dc-f100c9ae3789 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/precise.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/media/natty/home/gruber/ubuntu-kvm/tmpW8gSGB.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=18,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:0e:da:9b,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Here's the network portion of the virtual machine description in /etc/libvirt/qemu/quantal.xml

    <interface type='bridge'>
      <mac address='52:54:00:b0:8e:aa'/>
      <source bridge='br0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

According to this link, for performance and reliability, it may be best to set the network device model to virtio, you can do this by in the virt-viewer by pressing the i button, going to the NIC setting, and setting the "Device model" to virtio. You could also add this to the XML above by adding the line:

      <model type='virtio'/>

In Summary

All this took on 12.04 was:

1. Installing virt-manager, bridge-utils, qemu-kvm, and related packages
2. Make sure each user wishing to use kvm are in the libvirtd group.
3. Defining /etc/network/interfaces as above (which match the quoted article)
4. Reboot, making sure Ethernet is plugged in and wireless (if any) is off.
5. Either run kvm against an image directly with, e.g. -device e1000,netdev=tunnel -netdev tap,id=tunnel,ifname=vnet0, or create a virtual machine with virt-manager, specifying network Bridge br0 under the Step 4->Advanced Options panel.

No further changes were needed to networking, capabilities, templates, or configurations.

To expose a service in your new guest to the Internet you should:

1. Prepare and configure any firewall service you will need.
2. Either assign a static address in your guest configuration or in your DHCP service.
3. If you are using a NAT router open a port for the service you are implementing directing it to the guest's IP address.

Remember to test and re-enable the firewall service for your host computer. It may need any entry to forward traffic to the guest.

See https://help.ubuntu.com/community/KVM/Installation, https://help.ubuntu.com/community/KVM/Networking, and https://help.ubuntu.com/12.04/serverguide/libvirt.html.

Answer 2

If the behavior you are seeing is host can access the guest, and guest can access the host, but the guest can't access other machines on the network or visa versa... probably the host's firewall is blocking access.

See: https://bugs.launchpad.net/ubuntu/+source/ufw/+bug/573461

Specifically, this section: "The final step is to disable netfilter on the bridge:

# cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
EOF

Answer 3

These are the two scripts I use to create a bridge for qemu-kvm.

First, let the host become a IP router.

Script ip-router.sh:

#!/bin/bash

internetinterface="eth0"

username=`whoami`

if [ "x$username" != "xroot" ] ; then

    echo    
    echo "You must be root in order to run this script..."
    echo    

    exit    

fi  

if [ "x$1" != "x" ] ; then
    internetinterface="$1"
fi  

if [ "x$1" == "xdel" ] || [ "x$2" == "xdel" ] ; then
    disable="1"
else
    disable="0"
fi  

if [ "$disable" == "0" ] ; then
    echo "Enabling IP forward and setting up masquerade NAT on interface $internetinterface"

    echo 1 > /proc/sys/net/ipv4/ip_forward

    iptables -t nat -A POSTROUTING -o $internetinterface -j MASQUERADE
else
    echo "Disable IP forward and setting down masquerade NAT on interface $internetinterface"

    echo 0 > /proc/sys/net/ipv4/ip_forward

    iptables -t nat -D POSTROUTING -o $internetinterface -j MASQUERADE
fi  

Then, create the tun-tap interface and bridge it with your default interface (usually the one with an Internet connection).

Script create-qemu-bridged-tuntap.sh:

#!/bin/bash

bridgename=br0
tapinterface=tap0
outinterface=eth1

if [ "x$1" != "x" ] ; then
    outinterface="$1"
fi  

ifaces=`awk -F: '{print $1}' /proc/net/dev | tail -n +3`
iffound="0"

for i in $ifaces
do  
    if [ "$outinterface" == "$i" ] ; then
            iffound="1"
    fi  
done

if [ "$iffound" == "0" ] ; then
    echo
    echo "Can't find the output interface."
    echo
    exit 1
fi  

outifaceip=`ifconfig | grep -A1 $outinterface | tail -1 | awk -F: '{print $2}' | awk '{print $1}'`
outifaceiptokens=`echo $outifaceip | awk -F \. '{print NF}'`

if [ "$outifaceiptokens" != "4" ] ;  then
    echo
    echo "The selected output interface $outinterface doesn't seem to have a valid IP address."
    echo
    exit 1
fi  

hostaddress="192.168.1.1"
guestaddress="192.168.1.95"

sudo tunctl -t $tapinterface

sudo brctl addbr $bridgename
sudo brctl addif $bridgename $tapinterface

sudo ip link set $bridgename up
sudo ip addr add $hostaddress/24 dev $bridgename

sudo route add -host $guestaddress dev $bridgename
sudo parprouted eth1 $bridgename

sudo ~/scripts/ip-router.sh $outinterface

I use these scripts daily, so they should work well also for you. You'll have to install some package in order to have all of this working. Using:

dlocate `which COMMAND`

you can see which package is required to have COMMAND. For example to see which package is required to have brctl, simply run:

dlocate `which brctl`

and you'll have:

bridge-utils: /sbin/brctl

Using the same approach for all the commands in these scripts, you should (at least) run this aptitude command line:

sudo aptitude install dlocate iproute parprouted iptables uml-utilities bridge-utils net-tools

Finally, you can launch the main script (as a normal user):

#> create-qemu-bridged-tuntap.sh eth0
Set 'tap0' persistent and owned by uid 0
Enabling IP forward and setting up masquerade NAT on interface eth0

Running ip addr you should see a br0 interface with IP address 192.168.1.1, as specified inside the create-qemu-bridged-tuntap.sh script:

#> ip addr
8: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 36:76:ee:d6:63:b2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 scope global br0

This is the host address as seen by the guest. Conversely, the guest will have as IP address 192.168.1.95 (again, this can be easily changed inside the main script).

Now, using virt-manager, you just have to setup your guest nic to use br0 as physical interface.

Inside the guest, you just have to give to eth0 an IP address of 192.168.1.95 and everything should run fine.

Slackware13:~> ifconfig 
eth0      Link encap:Ethernet  HWaddr 52:54:00:F7:6A:78  
          inet addr:192.168.1.95  Bcast:192.168.1.255  Mask:255.255.255.0