I have 3 hosts that are connected via Wireguard mesh with each others. I want all three to have the same layer 2 traffic (it's about disaster recovery).
So far, I've setup a bridge on each, connected to a vxlan network. Things worked quite well, but vxlan explicitly forbids fragmentation, so when I connect the vxlans to my bridges, the bridges lower their MTU to 1500 bytes minus wireguard headers minus vxlan headers. I cannot allow my bridges to have lower than 1500 MTUs since it would imply that everything connected to those bridges would also have lower MTUs, including all virtual machines.
Reading a bit, I've replaced vxlan with geneve which allows fragmentation. Now, I can have my bridges with a 1500 MTU, geneve interfaces with 1500 MTU, and wireguard fragments geneve packets for me, which works quite well as long as I have two hosts only.
The problem is routing for me:
- vxlan is virtually a distributed switch, so there are no network loops
- geneve looks like a point-to-point protocol, so connecting three hosts togheter made a nice network loop with packet loss
I am quite sure I didn't understand how geneve works, since it's supposed to be "as good or better" than vxlan.
How do one configure a geneve interface with multiple remotes ?
On vxlan, you'd just do something like the following to "populate" the distributed switch fdb table and make vxlan aware of remote hosts:
bridge fdb append to 00:00:00:00:00:00 dst 10.20.30.1 dev vxlan0
bridge fdb append to 00:00:00:00:00:00 dst 10.20.40.1 dev vxlan0
But on geneve, I only can add one remote to my interface:
ip link add name geneve0 type geneve id 1000 remote 10.20.30.1
ip link set up geneve0
So I actually create two geneve interfaces (one per remote), and connect them to my brige on each host, which obviously is not the way it should work.
Is there a better way to do a 3-node distributed network using geneve ?
