Howdy!
This is how one is greeted, when opening the Rancher Management UI. Usually feeling very comfortable when using this awesome tool, we were having a problem with interhost communication in our cloud environment. It was very unclear, what was going on, since alle hosts could ping each other, but health checks failed. A clear tell, that something is not in order.
After messing around with iptables and a lot of trial and error, we could tackle this issue. Let’s start with a short roundup of what was going wrong:
Interhost connectivity and security is achieved through the ipsec rancher infrastructure. That basically means that every host is running one of those wonderful ipsec containers, exchanging a pre shared key via rancher communication. They form an overlay network, which lives in 10.42.0.0 subnet with default settings. To ensure that packets are routed correctly between hosts the rancher agent masquerades outgoing packets with the ip of the host machine. Now the problems begin.
Our hosts are assigned a private address and at least one public address. Obviously, we wanted our packets to get routed via the internal IP address. The private address is just an alias on eth0 – the only available network interface. Until here nothing special. This is a very common configuration. So why the heck do they refuse to connect to each other. As always, the devil is in the details. When forwarding packets our hosts were correctly masquerading the packets and they did reach the other hosts, but the target ipsec container was receiving the public IP as source – not the private one, so it correctly refused to establish the connection. Finally a trace.
Our first solution was to put an own iptable rewrite group into the POSTROUTING chain, which was inserted before rancher’s routing rules. Basically, we just took the rancher rules and changed them from MASQUERADE to SNAT. That in fact solves the problem in a very hacky way. MASQUERADE is dynamically changing the source IP via its network configuration. We are not 100% sure, why the public instead of the private IP is used, but switching to a static address translation via SNAT instead of using MASQUERADE solved this issue. To get an impression, these are the rules, which restored connectivity:
iptables -t nat -A SNAT_TEST -s 10.42.0.0/16 ! -o docker0 -p tcp -j SNAT --to-source <private host ip>:1024-65535
iptables -t nat -A SNAT_TEST -s 10.42.0.0/16 ! -o docker0 -p udp -j SNAT --to-source <private host ip>:1024-65535
iptables -t nat -A SNAT_TEST -s 10.42.0.0/16 ! -o docker0 -j SNAT --to-source <private host ip>
Now this enforces the private IP address on outgoing packets to all destinations in the ipsec network (10.42.0.0, as mentioned before the default for rancher). Because of the hackiness of this solution, we don’t want to get into more detail, since there is a better one (yes, there is always something better). In fact we talked with the rancher support about this problem and they were reacting extremely swiftly, offering professional help. After two remote desktop sessions we found a solution that is way cleaner and a short and efficient check, if this problem occurs.
So short and yet so powerful:
ip route get <private ip of the target host>
This will show you how the source IP of the packet looks like after MASQUERADING. Finally, this can be changed with the following statement
ip route add <private ip of the target host> via <private host ip>
Unfortunately, you have to add a rule for every target. And one disadvantage for both solutions: These changes are not persisted per default, since it is possible to lock yourself out, when applying rules that block ssh access, too 🙂
If you know a better way to configure the network interface properly to avoid this mess, let us know.
Do you want to see our support ticket? Have a look here