Setting up a bridge for host and vm

Question

I'm having quite a lot of problems setting up a network bridge for both my host and my vm to use.

The host (os of the server) is Ubuntu 20.04.4 LTS. The VM (using KVM and virt-manager) is TrueNAS Core (FreeBSD). I have one ethernet port on my server (on the motherboard) that I want to use for both my os and the vm. The os needs to get the ip of 192.168.2.15 with dns set to 1.1.1.1 and the vm needs to get the ip of 192.168.2.16. They both need to be accessible from "the outside". So, as far as I understand, I need to setup a network bridge.

Well it's all going wrong. I started with this video but with no success. Then I tried this video but also with no success. Then I tried this guide but again with no success. Then I turned to SO and tried these (1, 2) but all with no success.

Evertime there's something wrong. Most of the time DNS is not working, or the ip addresses are messed up (completely different subnet), or the os has the correct ip but can't ping any other devices (not 1.1.1.1 but also not 192.168.2.79 which is my static-ip-ed win10 computer) or /etc/network/interfaces is ignored (the file didn't exist yet). And most of the time people suggest using sudo /etc/init.d/networking restart or similar using systemd but evertime I get the message that networking doesn't exist which I find peculiar because everyone else does have it.

It's just all a mess and I can't get it to work. I would really appreciate some help!

The file /etc/network/interfaces does currently not exist. ifconfig -a output:

enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.15  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::8506:f97f:df7c:f66b  prefixlen 64  scopeid 0x20<link>
        ether 18:c0:4d:9b:3c:12  txqueuelen 1000  (Ethernet)
        RX packets 10390  bytes 3524106 (3.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2534  bytes 330756 (330.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfc600000-fc61ffff
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 7172  bytes 712208 (712.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7172  bytes 712208 (712.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:06:8d:b0  txqueuelen 1000  (Ethernet)
        RX packets 142  bytes 12152 (12.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40  bytes 4136 (4.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
virbr0-nic: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 52:54:00:06:8d:b0  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc54:ff:fee6:a823  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:e6:a8:23  txqueuelen 1000  (Ethernet)
        RX packets 142  bytes 14140 (14.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 939  bytes 55015 (55.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

EDIT: inside /etc/netplan, there is one file: 01-network-manager-all.yaml. Contents:

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

I don't care about network manager. If there is some solution that works but breaks network manager, I'm totally fine with that.

You seem to be creating your virbro-nic bridge on the VM. You should be creating the bridge on the host and then asking the VM to use it. I see that both askubuntu references point to old answers of mine, but from back when /etc/network/interfaces was being used. See if you can extract what to do from my question here, as it'll be many hours before I can write an answer. — Doug Smythies, Jun 06 '22 at 13:32
Please edit your question adding contents of /etc/netplan. I only use servers, and have /etc/netplan/01-netcfg.yaml which in turn uses networkd as renderer. I think desktop might use NetworkManager as renderer. I want to know what you have. Is it still /etc/netplan/00-installer-config.yaml from here ? — Doug Smythies, Jun 06 '22 at 13:57
@DougSmythies I'm doing everything on the os (Ubuntu). The VM is TrueNAS Core in which I do nothing regarding this bridging. So all the files I edit or settings I change are on the os. I'll edit the post to include info. — Cas, Jun 06 '22 at 14:23
It is not /etc/netplan/00-installer-config.yaml anymore. I used to have TrueNAS as the os with a Ubuntu server VM. Then I bought a GPU that I wanted to add to the Ubuntu instance but TrueNAS doesn't support pcie pass through. So I decided to reset everything and use Ubuntu (non-server; other reasons) as the os with a TrueNAS VM. I switched everything around. But I can't manage to set up networking like I could originally. — Cas, Jun 06 '22 at 14:33

Doug Smythies · Accepted Answer · 2022-06-09T18:18:31.143

I only have experience with networkd as the renderer not NetManager, so I don't know the value of this answer.

Save /etc/netplan/01-network-manager-all.yaml somewhere, so that you can revert if things do not work.

Create /etc/netplan/01-netcfg.yaml (gateway address assumed):

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp4s0:
      dhcp4: no
  bridges:
    br0:
      interfaces: [ enp4s0 ]
      dhcp4: no
      addresses: [192.168.2.15/24]
      gateway4: 192.168.2.1
      nameservers:
        addresses: [1.1.1.1]
      dhcp6: no
      link-local: [ ]
      parameters:
        stp: true
        forward-delay: 4

Do:

sudo netplan apply

Go to /etc/libvirt/qemu/networks. There should be a host-bridge.xml file. Save a copy:

sudo cp host-bridge.xml host-bridge.xml.original

If you do not have a host-bridge.xml file, my original was:

doug@s19:~/config/etc/libvirt/qemu/networks$ cat host-bridge.xml.original
<network>
    <name>host-bridge</name>
    <bridge name='br0'/>
    <forward mode="bridge"/>
</network>

Runs these commands:

virsh net-define host-bridge.xml
virsh net-autostart host-bridge
virsh net-start host-bridge
virsh net-list --all

In case of having issues and needing to undo, be aware of:

virsh net-destroy br0
virsh net-undefine br0

Example:

doug@s19:~/config/etc/libvirt/qemu/networks$ virsh net-list --all
 Name          State    Autostart   Persistent
------------------------------------------------
 host-bridge   active   yes         yes

Example:

doug@s19:~/config/etc/libvirt/qemu/networks$ sudo cat /etc/libvirt/qemu/networks/host-bridge.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit host-bridge
or other application using the libvirt API.
-->
<network>
  <name>host-bridge</name>
  <uuid>d474d859-8448-4055-9fc0-bb0b6cb83c34</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

Now, modify any existing VMs to use the bridge:

Note: There is probably a way to do this step using virt-manager, however I do not use it and do not know how.

Use virsh edit and change your interface definition lines to use bridging instead of what is currently being used. Something like this:

Before:

<interface type='network'>
  <mac address='52:54:00:06:8d:b0'/>
  <source network='default'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

after:

<interface type='bridge'>
  <mac address='52:54:00:06:8d:b0'/>
  <source bridge='br0'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

For creating a new VM, and depending on your method, something like:

virt-install -n serv-jj -r 8192 \
--disk path=/home/doug/vm/serv-jj.img,bus=virtio,size=50 \
-c jammy-live-server-amd64-2022-04-21.iso \
--network bridge=br0,model=virtio,mac=52:54:00:27:1c:6e \
--graphics vnc,listen=0.0.0.0 --noautoconsole -v --vcpus=4 --cpu SandyBridge

Examples with 3 VMs running:

doug@s19:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 3c:7c:3f:0d:99:83 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:7c:3f:0d:99:83 brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.136/24 brd 192.168.111.255 scope global dynamic br0
       valid_lft 81610sec preferred_lft 81610sec
4: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:22:2f:dc brd ff:ff:ff:ff:ff:ff
5: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:60:ea:3e brd ff:ff:ff:ff:ff:ff
6: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:60:ea:5e brd ff:ff:ff:ff:ff:ff
doug@s19:~$ networkctl
IDX LINK   TYPE     OPERATIONAL SETUP
  1 lo     loopback carrier     unmanaged
  2 enp3s0 ether    enslaved    configured
  3 br0    bridge   routable    configured
  4 vnet0  ether    carrier     unmanaged
  5 vnet1  ether    carrier     unmanaged
  6 vnet2  ether    carrier     unmanaged
doug@s19:~$ brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.3c7c3f0d9983       no              enp3s0
                                                        vnet0
                                                        vnet1
                                                        vnet2

Not covered herein: How to set the static address of 192.168.2.16 on the VM, because the VM was stated to be FreeBSD and I do not know how.

References:
https://netplan.io/examples/#configuring-network-bridges
KVM Network Bridge to assign Static IP

Hey @DougSmythies thank you very much for the guide. I'm having a problem: inside /etc/libvirt/qemu/networks, there is no host-bridge.xml file. Only an autostart folder and a default.xml file. I did touch host-bridge.xml but virsh says "Document is empty" so that didn't work. Help! — Cas, Jun 08 '22 at 20:20
@Cas : Answer edited, adding a listing of the original file. — Doug Smythies, Jun 08 '22 at 23:31
I've continued following the guide and when I do virsh net-list --all, I indeed see host-bridge in the list (even after reboot). And the host-bridge.xml file has indeed changed its contents. In virt-manager I set the nic of the VM to the bridge and saw that the XML contents matched what you showed so everything is good. However, br0 isn't showing up in ip addr, networkctl or brctl show. The os can't reach anything and the VM won't start saying that br0 can't be found. So a new problem... — Cas, Jun 09 '22 at 11:21
I found out that networkd wasn't running. So I did sudo systemctl enable systemd-networkd.service and also start and rebooted. networkctl now shows actual statuses under OPERATIONAL instead of N/A. However, br0 is still not popping up anywhere except 'host-bridge' in virsh net-list --all. Os still can't connect and VM still can't start — Cas, Jun 09 '22 at 11:33
Hey we're definitely making progress! br0 is now popping up in ip addr, networkctl and brctl show. However, in networkctl, it shows br0 as "no-carrier" and "configured" which does not match up with your output. Also, in brctl show, it shows no interfaces while in your output it shows the os ethernet and the VM ethernets. We're making progress though! — Cas, Jun 09 '22 at 17:54
WE DID IT!! WE DID IT!! I noticed in the ip addr output that my ethernet port was referenced to by enp5s0 instead of 4. I don't know what made the number go up suddenly but I edited the netplan file to change the 4 to 5 and did the apply command, rebooted and IT WORKED!! I want to thank you very very much. I truely appreciate it. — Cas, Jun 09 '22 at 18:01
Please add the sudo netplan apply command to your answer. That way me or anyone else landing on this page will have the complete guide from your answer without needing to look in the comments. — Cas, Jun 09 '22 at 18:04

score 0 · Answer 2 · answered Jan 20 '23 at 17:38

0

this might help, if it doesn't get blocked, It is not thoroughly tested and some more tweaking is necessary, welcome to update it, just let me know of any changes and fixes you make.

Probabably no need to generate the entire file , it was just easier at the time (read the code you will see what you mean) also the user name sam and ip addresses have to be changed.

https://github.com/wanfuse123/bridgeNET.git

answered Jan 20 '23 at 17:38

user1666519

1

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – Pilot6 Jan 23 '23 at 11:03

Setting up a bridge for host and vm

2 Answers2

Linked