cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David <d...@ntso.com>
Subject SNAT issue
Date Wed, 21 May 2014 17:18:28 GMT
Hello

Cloudstack VM Issue, no outgoing access

There is an issue in cloudstack where a VM cannot gain outgoing access to 
the internet. This is a
blocking issue since Cloudstack users cannot gain Internet access to do 
patch updates, download
software, etc.

Environment:

Cloudstack version 4.3
System VM from January 2014
CentOS6.5 for all servers
4 servers (management server, 2 compute nodes, and a server that serves both 
primary and secondary
storage)
Advanced networking zone with 1 zone, 1 pod, 1 cluster, 2 hosts
2 NICs on the compute node hosts
- 1 NIC hosts the management and storage
- 1 NIC hosts the public and guest networks
Primary and Secondard storage use NFS
Management network is 172.16.10.x/24
Guest networks are 10.0.0.x/24
Public network uses a /24 public range


Here is how to reproduce the prove the issue.

1. Create a new VM using a new isolated network
2. Create the egress rules to allow 10.0.0.0/24 out for everything
3. Create a port map rule for SSH to the new VM and also allow external 
acess to SSH
4. Login to the console on the new VM
5. Ping google.com, see 100% packet loss.
6. Login to the console on the virtual router for this network and type this 
command:

Ping results
[root@NewTestVM ~]# ping google.com
PING google.com (74.125.225.128) 56(84) bytes of data.
^C
--- google.com ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time
6127ms
[root@NewTestVM ~]#



All the iptables rules look as they should, especially look at the SNAT rule 
in the nat table for
POSTROUTING. But, still can't ping. How do we know there isn't some other 
problem in the
network. Well, first we were able to SSH to the new VM externally from the 
Internet to do the ping.


Second, we found that if we add a second identical SNAT rule, now everything 
works.

Okay so now add the second SNAT rule like this;

    iptables -t nat -A POSTROUTING -j SNAT –t-source 68.70.175.6

Yes, pinging google.com still does not work, losing 100% of packets..


Now, Pinging from the new VM works like a charm. What is going on??
[root@NewTestVM ~]# ping google.com
PING google.com (74.125.225.132) 56(84) bytes of data.
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=1 ttl=52 
time=2.12 ms
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=2 ttl=52 
time=2.09 ms
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=3 ttl=52 
time=2.22 ms
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=4 ttl=52 
time=2.24 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3556ms
rtt min/avg/max/mdev = 2.095/2.172/2.249/0.064 ms
[root@NewTestVM ~]#


The first SNAT rule should have worked but it doesn't. WE've proven, I 
think, that everything else is
setup correctly by being able to access the VM externally via SSH and by 
simply adding the SNAT rule
to the virtual router. Why doesn't the first rule work.


It seems cloudstack is doing exactly what it is supposed to do, adding the 
SNAT rule. So, it seems the
actual problem is with either iptables or with the debian kernel for the 
system VM. I think we are using
the most recent system VM.

Note, neither restarting the new network nor cleaning it makes any 
difference. It just seems having the
one SNAT rule will not work. I've done all the tcpdump debugging, that is 
how I found this issue to
begin with. Tcpdump shows that the ping/icmp packets go through the virtual 
router unaltered using
the private IP address of the VM (10.0.0.x). After the second SNAT rule is 
added tcpdump shows then
that the ping/icmp packets do get translated.

I am totally floored by these results. How can iptables or the debian kernel 
not recognize the first
SNAT rule and then recognize the second one??


Okay, so before someone tells me that this is not a cloudstack issue and to 
find a resolution else,
cloudstack depends on this technique for isolated networks. If guest users 
can't make their Vms work
(note, they don't have access to the virtual routers like I do), isolated 
networks are broken and
cloudstack is not usable.


I have screen shots i can send if it will help.


Mime
View raw message