Using OpenvSwitch with DPDK in Ubuntu 16.04

Question

I am trying to setup openvswitch with DPDK in Ubuntu 16.04 using the openvswitch-switch-dpdk package. I am using an HP DL380 Gen-8 server with an Intel 82599ES Dual Port 10G card.

I can bind the ports on the Intel card to the DPDK driver (have tried with both vfio-pci and uio_pci_generic) successfully, but cannot add these interfaces to the OVS bridge, as I keep getting the error could not open network device dpdk0 (No such device). I am following the procedure from this link.

Any help in this regard would be appreciated.

Server output

• VT-d is enabled in the BIOS and iommu enabled in grub

   root@kvmtwo:~# cat /proc/cmdline
  
   BOOT_IMAGE=/vmlinuz-4.4.0-22-generic root=/dev/mapper/kvmtwo--vg-root ro iommu=pt intel_iommu=on
  

• Hugepages is enabled

  root@kvmtwo:~# cat /proc/meminfo |grep -i huge
  AnonHugePages:    135168 kB
  HugePages_Total:    1024
  HugePages_Free:      512
  HugePages_Rsvd:        0
  HugePages_Surp:        0
  Hugepagesize:       2048 kB
  

• Two 10G ports on an Intel 82599 NIC are configured to use the uio_pci_generic driver

  root@kvmtwo:~# dpdk_nic_bind -s
  Network devices using DPDK-compatible driver
  
  ============================================
  
  0000:07:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe
  0000:07:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe
  
  
  
  Network devices using kernel driver
  
  ===================================
  
  0000:03:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=uio_pci_generic
  0000:03:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=uio_pci_generic *Active*
  0000:03:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=uio_pci_generic
  0000:03:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=uio_pci_generic
  
  
  
  Other network devices
  
  =====================
  
  <none>
  

• Contents of /etc/default/openvswitch-switch and /etc/dpdk/dpdk.conf

  #/etc/default/openvswitch-switch
  
  DPDK_OPTS='--dpdk -c 0x1 -n 4 -m 4096,4096 --vhost-owner libvirt-qemu:kvm --vhost-perm 0664'
  
  
  
  /etc/dpdk/dpdk.conf
  
  NR_2M_PAGES=1024
  
  NR_1G_PAGES=4
  

• The openvswitch is running and using the DPDK libraries

  root@kvmtwo:~# ovs-vsctl show
  44ea2456-b7dc-4b76-a7da-222483ea08af
  ovs_version: "2.5.0"
  
  
  
  root@kvmtwo:~# ps -ef|grep ovs|grep dpdk
  
  root      1629     1  0 14:47 ?        00:00:00 ovs-vswit... --dpdk -c 0x1 -n 4 -m 4096,4096 --vhost-owner libvirt-qemu:kvm --vhost-perm 0664 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
  root      1630  1629  0 14:47 ?        00:00:03 ovs-vswitchd --dpdk -c 0x1 -n 4 -m 4096,4096 --vhost-owner libvirt-qemu:kvm --vhost-perm 0664 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
  

• I can add the OVS bridge:

  root@kvmtwo:~# ovs-vsctl show
  44ea2456-b7dc-4b76-a7da-222483ea08af
  Bridge "ovsbr0"
  Port "ovsbr0"
  
      Interface "ovsbr0"
  
          type: internal
  
  ovs_version: "2.5.0"
  

• I cannot add the 10G ports with DPDK driver to the OVS bridge.

  root@kvmtwo:~# ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
  ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd log for details.
  
  
  
  root@kvmtwo:~# ovs-vsctl show
  44ea2456-b7dc-4b76-a7da-222483ea08af
  Bridge "ovsbr0"
  Port "dpdk0"
      Interface "dpdk0"
          type: dpdk
          error: "could not open network device dpdk0 (No such device)"
  Port "ovsbr0"
      Interface "ovsbr0"
          type: internal
  ovs_version: "2.5.0"
  
  root@kvmtwo:~# tail /var/log/openvswitch/ovs-vswitchd.log
  2016-06-08T10:05:40.251Z|00054|bridge|WARN|could not open network device dpdk0 (No such device)
  

Answer 1

TL;DR:

look at syslog and ovs-log when starting OpenvSwitch with dpdk what goes wrong on dpdk & port initialization.

Detail:

your setup looks good so far compared to what I usually see. Given what you reported the most likely issue is right where the last error message points you to "See ovs-vswitchd log for details." or in the Syslog where most of the EAL messages go.

At the time you add the port you likely just see that it can't find it. What will be more interesting for you is the log around the time when you (re-)start openvswitch. There you should see it discovering and initializing the ports dpdk0 and dpdk1 - or if not, what errors are associated.

As a reference I uploaded my log, split into three columns for command, ovs-log and syslog. I'd expect that something around my line 54 is not working right in your case.

should be something like this:

ovs-ctl[3560]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
ovs-ctl[3560]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
ovs-ctl[3560]: EAL:   PCI memory mapped at 0x7f2140000000
ovs-ctl[3560]: EAL:   PCI memory mapped at 0x7f2140200000

Note: there is also this Ubuntu guide to help you https://help.ubuntu.com/16.04/serverguide/DPDK.html.