After struggling for over 20 hours, I wanted to share the results of my investigation regarding very poor Internet upload erformance.

#Setup

  • Proxmox Server with a Single 10GbE NIC
  • OPNsense VM on Proxmox
  • OPNsense uses VirtIO NICs tied to the 10GbE Linux Bridge
  • upstream Gateway is a OpenWRT router with 1GbE uplink
  • Zyxel XS1930 Switch connecting Proxmox Host and Gateway

#Problem Internet download speeds are fine (900Mbit/s) but upload speeds are not (5-15MBit/s instead of 50MBit/s)

#Solution Various OPNsense tunables (configured for 8 CPU cores)

  • hw.ibrs_disable = 1
  • net.isr.maxthreads = -1
  • net.isr.bindthreads = 1
  • net.isr.dispatch = deferred
  • net.inet.rss.enabled = 1
  • net.inet.rss.bits = 6
  • kern.ipc.maxsockbuf = 16777216
  • net.inet.tcp.recvbuf_max = 4194304
  • net.inet.tcp.recvspace = 65536
  • net.inet.tcp.sendbuf_inc = 65536
  • net.inet.tcp.sendbuf_max = 4194304
  • net.inet.tcp.sendspace = 65536
  • net.inet.tcp.soreceive_stream = 1
  • net.pf.source_nodes_hashsize = 1048576
  • net.inet.tcp.mssdflt = 1240
  • net.inet.tcp.abc_l_var = 52
  • net.inet.tcp.minmss = 536
  • kern.random.fortuna.minpoolsize = 128
  • net.isr.defaultqlimit = 2048

Enabling Multiqueue in Proxmox for the VirtIO NICs

(binary stepping, 1 Queue for 2 cores, 2 Queues for 4 cores, 3 Queue for 8 cores ect, total amount of all Queues mustn’t be greater then the VMs CPU cores)

Enabling Flow Control on all involved Network devices

  • Proxmox hardware NIC: ethtool -K nic0 rx on tx on
  • OpenWRT lan interfnace:

uci set network.lan.txpause='1'

uci set network.lan.rxpause='1'

uci commit

reload_config

  • Zyxel Switch: Port -> Port Setup

Checked all Ports

Enabling Port Buffering

Zyxel Switch: Port -> Port Buffer

Checked the Port with the Gateway

#Reason

The Main reason for this problem seems to be the down-stepping of 10Gbit traffic to 1Gbit devices. Without Flow control enabled on all involved devices, the sending rate can’t be adjusted. But without enabling Port Buffering, the Switch won’t allocate resources for adjusting the traffic flow rate for slower devices.

This Problem should only affect people who use devices with different link speeds on the same switch.

  • litchralee@sh.itjust.worksEnglish
    0·
    3 hours ago

    Given that your original problem was related to WAN upload performance, why did your investigation lead you to Ethernet flow-control? An ISP connection generally deals in packets at Layer 3 (“network”, eg IP) of the OSI model, whereas Ethernet is a Layer 2 (“data-link”) layer technology.

    If there is a bottleneck at your WAN modem, then that will cause congestion at layer 3, but Ethernet flow-control can only deal with congestion that exists at layer 2. What has likely happened is that you have configured your gateway so that congestion at layer 3 is mirrored onto your layer 2 LAN. And if flow-control is enabled, then that would result in back-pressure propagating back to your VMs. Your VMs will then slow down their layer 2 rate, which conveniently forces the layer 3 traffic to also slow down.

    This is an incredibly round-about and inefficient way to do traffic shaping. You should not configure a network so that L3 and L2 issues bleed into each other. A major consequence of using flow-control in this way is that it reduces the capacity of your LAN, even for traffic that isn’t going out to the WAN.

    The customary approach for keeping L2 and L3 separate is to perform traffic shaping solely at the threshold where your LAN meets the bottleneck. This would be your gateway, since after the gateway would be the WAN (50 Mbps upload). The gateway would be configured with some sort of QoS feature so that certain L3 packets are selectively dropped.

    You cannot do effective L3 traffic shaping without dropping packets. In fact, all competent L3 protocols expect dropped packets in order to slow down their data rate: SCTP and TCP have their own exponential congestion control mechanism, UDP simply doesn’t guarantee delivery whatsoever, and QUIC has its own as well. Simply put, all L3 protocols only understand one signal that tells them to slow down, and it is to drop a few packets. They will adjust accordingly.

    TL;DR: please try OpenWRT QoS instead

    • Victor@lemmy.worldEnglish
      0·
      2 hours ago

      This is mostly going over my head, but I read all of it and it is very well written. Objective and factual (it seems to me). No belittling, just genuinely trying to deliver helpful criticism and explanation.

      🤝🙇‍♂️

      Good on you for being a good fellow human.