3

I have an Ubuntu 14.04 server with one NIC. The MAC address on this NIC is assigned (by the data centre's gateway) one IP (X.X.X.100) in one subnet (X.X.X.0/24), with the gateway being in the same subnet (X.X.X.1), and an additional IP range from a completely different subnet (Y.Y.Y.0/28).

I've set up the server with both X.X.X.100 and Y.Y.Y.1 on the same interface (eth0), and the server can contact and be contacted by the rest of the Internet - on both IP addresses.

I have some virtual machines that use the IP addresses from the second subnet. I assume they must use the host's second IP address Y.Y.Y.1 as their gateway address (as they won't be able to reach the data centre's gateway address due to it being in a different subnet - as far as I understand). All the IP addresses I have, X.X.X.100 and Y.Y.Y.0/28 are statically routed on the data centre's gateway, so any request destined for them will arrive at my server's NIC. It will also only accept requests from the MAC address on that NIC, not the MAC addresses of the VMs.

How do I set up the host so that all requests to or from the Y.Y.Y.0/28 subnet are routed via the data centre's gateway X.X.X.1?

As far as I know, I cannot simply expand the netmask, as this could lead to not being able to contact other servers on the net.

Here's the server's /etc/network/interfaces

auto lo
iface lo inet loopback

auto  eth0

iface eth0 inet static
  address   X.X.X.100
  netmask   255.255.255.0
  gateway   X.X.X.1
  dns-nameservers 8.8.8.8 8.8.4.4

iface eth0 inet static
  address   Y.Y.Y.1
  netmask   255.255.255.240

In /etc/sysctl.conf I have enabled this line:

net.ipv4.ip_forward=1

The VMs use Y.Y.Y.n (where n>1) as IP address and Y.Y.Y.1 as gateway.

With the above setup, the VMs can contact the host on both its IP addresses, but nothing else. I think the VMs have trouble finding the MAC address of the ISP's gateway X.X.X.1:

$ arp -an
? (X.X.X.1) at c8:60:00:5e:bd:e0 [ether] on eth0
? (Y.Y.Y.1) at <incomplete> on eth0

However strangely, they can sometimes...

$ arp -an
? (X.X.X.1) at c8:60:00:5e:bd:e0 [ether] on eth0
? (Y.Y.Y.1) at cc:e1:7f:07:e0:af [ether] on eth0

These two examples where taken only a few minutes apart. I have a feeling that pinging the VM, especially pinging from the VM speeds up the process of making it get the MAC address, although this may be purely incidendal.

When it "has" the MAC address og X.X.X.1, everything actually works! I can access the Internet from the VMs and access them from the internet using their Y.Y.Y.n address. But it doesn't work all the time - occasionally it "forgets" the MAC address of Y.Y.Y.1 again, and becomes inaccessible.

I'm actually surprised it lists Y.Y.Y.1 in the ARP list at all - I thought that was only for devices in the same subnet. My goal was to make them just use the host as gateway address (Y.Y.Y.1) instead and avoid the problem by letting the host to all the communication on X.X.X.0/24.

Is my configuration a sensible one?

Are anyone else using this configuration?

What possible reasons could there be for the weird "forgetfulness"?

UPDATE:

I've tried forcing the VMs to use the datacentre's gateway's MAC address directly instead of the host's for the IP address they use as gateway - Y.Y.Y.1 - but the problem persists:

$ arp -an
? (X.X.X.1) at <incomplete> on eth0
? (Y.Y.Y.1) at c8:60:00:5e:bd:e0 [ether] PERM on eth0
raw@test-server:~$ sudo arp -s Y.Y.Y.1 cc:e1:7f:07:e0:af
raw@test-server:~$ arp -an
? (X.X.X.1) at <incomplete> on eth0
? (Y.Y.Y.1) at cc:e1:7f:07:e0:af [ether] PERM on eth0
raw@test-server:~$ ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From Y.Y.Y.3 icmp_seq=1 Destination Host Unreachable
From Y.Y.Y.3 icmp_seq=2 Destination Host Unreachable
From Y.Y.Y.3 icmp_seq=3 Destination Host Unreachable
From Y.Y.Y.3 icmp_seq=4 Destination Host Unreachable

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3001ms
pipe 3

Only when there's an ARP entry for X.X.X.1 does it work properly, but I can't add it manually with arp -s (error message: SIOCSARP: Network is unreachable), and I don't know what makes it randomly appear or disappear:

$ arp -an
? (X.X.X.1) at cc:e1:7f:07:e0:af [ether] on eth0
? (Y.Y.Y.1) at cc:e1:7f:07:e0:af [ether] PERM on eth0
raw@test-server:~$ ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=5.92 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=6.13 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=6.13 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=6.13 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3008ms
rtt min/avg/max/mdev = 5.929/6.083/6.138/0.118 ms

A couple of minutes after this, it went away again:

$ arp -an
? (X.X.X.1) at <incomplete> on eth0
? (Y.Y.Y.1) at cc:e1:7f:07:e0:af [ether] PERM on eth0
Improve this question
12
  • You use "gateway" in a confusing manner, first it's what's assigning the DHCP addresses to your host, then you're using the host itself as gateway. So your host is X.X.X.100, the gateway is apparently X.X.X.1? Does X.X.X.1 also have an address in the Y.Y.Y.0/28 range? When your host contacts other hosts on the internet, does it use its X.X.X.100 address or the Y.Y.Y.1 address? The latter seems unlikely... Are the VMs connected via a bridge to the host? Commented Oct 22, 2015 at 12:34
  • Apologies for the confusion. X.X.X.1 is the host's gateway. It only has that one IP address and is not controlled by me but by the data centre. My server (the host of the VMs) has IP address X.X.X.100, so it can connect to the gateway. At the same time, it needs to act as the gateway for the VMs, so it also has Y.Y.Y.1, which is within their subnet. Perhaps I'm using the wrong word - maybe I should rather call it router? I have edited the question to make it a bit more clear. Commented Oct 22, 2015 at 13:10
  • If your VMs have separate IP addresses from what the data centre uses, then presumably there needs to be some form of SNAT applied to outgoing connections from those VMs, as the Y.Y.Y.0/28 addresses won't be routed by the data centre. The VMs need to have Y.Y.Y.1 as the gateway, and the host needs to perform the SNAT translation on connections coming from Y.Y.Y.0/28 going out to X.X.X.1 so that as far as X.X.X.1 is concerned, all traffic seems to be coming from X.X.X.100. In short you need to describe exactly how the VM network interfaces are configured : bridge/NAT/whatever Commented Oct 22, 2015 at 13:35
  • OK, that wasn't clear either. Both the X.X.X.100 address as well as the Y.Y.Y.0/28 subnet was given to me by the data centre. All the addresses are routed to the NIC of the physical server (host), and I've verified that this works fine as I can reach the server from both X.X.X.100 and Y.Y.Y.1. NAT is not necessary, nor will it suffice, as all VMs should be able to make full use of their unique IP addresses, just like a physical server. The traffic should go via from the VM Y.Y.Y.2 should go via Y.Y.Y.1 / X.X.X.100 to X.X.X.1, but should appear to be from Y.Y.Y.2. Commented Oct 22, 2015 at 15:24
  • Forgot to mention, as of now, I've configured the VMs VirtualBox to use bridging, connecting to eth0 on the host. As far as I've understood this is the only setting I could use in order make the VMs appear as physical servers with one public IP each. But I could of course be wrong, so that needs to be taken into account :) If there are other ways to do it, I am open to suggestions, as long as it is something I can do on either the host server or the VMs. I can't do anything about the datacentre's gateway machine X.X.X.1. That's configured that way and they do not want to change it. Commented Oct 22, 2015 at 15:31

1 Answer 1

Reset to default
1

As your data center operators seem not quite understand how routing works, (you need a gateway that is inside your subnet for things to work normally), you need to add a static dummy arp entry with the MAC address of the gateway and an IP address that is inside your subnet (and not used!).

Use one of the Y.Y.Y.0/28 addresses that is not in use, and pretend that it is the gateway (perhaps it should be Y.Y.Y.1 anyway). Use that IP address as the gateway in the routing table. Now assign a static arp entry to that address using arp -s Y.Y.Y.x c8:60:00:5e:bd:e0 so that your VM doesn't bother with arp requests (the hardware address is already in the ARP table) and sends it to the gateway.

Explanation:

Normally when a system wants to contact another host on its subnet, that system will use ARP to find out what the MAC address is that has the requested IP address. The host will then respond to the ARP request; if it doesn't, you finally get "host unreachable" errors.

Now you have a host (gateway router) that is not in your subnet for some reason. As all you have to do is send IP packets for destinations outside your subnet via that router, you can use a fake IP address and create a static ARP table entry for that router with that fake IP address. Now all traffic that needs to go to (or through) that router will be addressed to the correct MAC address without your system having to try to discover that MAC address via ARP broadcasts.

I've managed to configure a printer gateway box that way: enter its MAC address with a free local IP address as a static ARP entry, then telnet to that address and configure it properly.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.