cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiradeep Vittal <Chiradeep.Vit...@citrix.com>
Subject Re: Advanced Network - SNAT not working
Date Mon, 16 Sep 2013 20:00:40 GMT
My theory was that by logging into the VR and running commands, you might
have inadvertently changed something.
What is the output of
iptables-save after the restart?

On 9/16/13 12:04 PM, "Noel Kendall" <noeldkendall@hotmail.com> wrote:

>That has worked Chiradeep. What could have caused this problem? Is it
>somethingthat should be fixed?
>Thanks for the simple and rather effective suggestion!
>Noel
>
>> From: Chiradeep.Vittal@citrix.com
>> To: users@cloudstack.apache.org
>> Subject: Re: Advanced Network - SNAT not working
>> Date: Mon, 16 Sep 2013 18:00:29 +0000
>> 
>> Suggest that you stop and start (not reboot) the router from the Admin
>>GUI.
>> 
>> On 9/16/13 5:26 AM, "Noel Kendall" <noeldkendall@hotmail.com> wrote:
>> 
>> >Jayapal, I did a ping test and traced as you suggested. tcpdump
>> >monitoring was done on the public facing interface of the VR.
>> >From within the VR, ping to public IP functions correctly, source
>>address
>> >is the public IP assigned to the VR.
>> >From within the guest, ping to same public IP, does not function,
>>source
>> >address is (as you suspected) the IP of guest on the guest network of
>>VR.
>> >Therefore, it must be: the SNAT rule in iptables in the VR is being
>> >bypassed... that is, the packets are being forwarded without SNAT being
>> >performed on them correctly.
>> >Noel
>> >
>> >> From: jayapalreddy.uradi@citrix.com
>> >> To: users@cloudstack.apache.org
>> >> Subject: Re: Advanced Network - SNAT not working
>> >> Date: Mon, 16 Sep 2013 05:14:53 +0000
>> >> 
>> >> Hi,
>> >> 
>> >> I think when the packets are going out the packets are NATed with
>> >>private ip, that can't reach back to router.
>> >> From the VR when you ping public network observe with what source ip
>> >>address the packet is going out and
>> >> From the guest VM when you access public n/w observe on VR with what
>> >>source ip the packet is going out.
>> >> In later case I think the source ip address is different.
>> >> 
>> >> Thanks,
>> >> Jayapal
>> >> 
>> >> 
>> >> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkendall@hotmail.com>
>> >>wrote:
>> >> 
>> >> > No other NAT. There is nothing but copper between the KVM host
>> >>machine and the ISP router.There is an L2/L3 switch that the packets
>> >>travel through. However, there is no forwarding in the switch,just
>> >>straight through. I've had a well-functioning V4.0.1 environment
>>running
>> >>on this same configurationin the past. What is new is the conversion
>>to
>> >>4.1 (which was a clean install).
>> >> > It's very mysterious, I have never seen anything like this before.
>> >>There are two other VRs, both having same issue.
>> >> > I will try your suggestion.
>> >> > Noel
>> >> >> Date: Sun, 15 Sep 2013 21:20:41 +0100
>> >> >> Subject: Re: Advanced Network - SNAT not working
>> >> >> From: msweet.dev@gmail.com
>> >> >> To: users@cloudstack.apache.org
>> >> >> 
>> >> >> This is mostly confusing that the packets are not seen on the VR
>> >>public
>> >> >> interface, seeing as other services are working.
>> >> >> If it was a local NAT issue then the packet would atleast get into
>> >>that
>> >> >> interface. Do you have any upstream devices providing NAT? Or any
>> >>other VR
>> >> >> with the issue?
>> >> >> 
>> >> >> It may be worth recreating the VR, by stopping and destroying it
>>and
>> >> >> creating another guest to start a fresh.
>> >> >> 
>> >> >> Marty
>> >> >> 
>> >> >> 
>> >> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall
>> >><noeldkendall@hotmail.com>wrote:
>> >> >> 
>> >> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell
in the
>>guest,
>> >> >>> while running a tcpdumpon the public i/f of the VR:
>> >> >>> - I can see the outbound packets going out- I do not see a
>>response
>> >>packet
>> >> >>> coming back in
>> >> >>> FYI there are no firewalls outbound from the KVM host. The
host
>> >>bridges vi
>> >> >>> CS networkingdirectly out on to the internet via a switch.
>> >> >>> Note that traffic from outside (ssh, web) can happily traverse
>>the
>> >>VR to
>> >> >>> the guest. I get the usualits working html page from the guest.
>> >>This tells
>> >> >>> me that there is nothing outbound from the VR thatis filtering
>> >>packets.
>> >> >>> Am truly stumped. This is mysterious indeed.
>> >> >>> From within the VR, can happily telnet to <www.xyz.com>
80 and
>> >>receive
>> >> >>> response.Only if packet came from guest and was forwarded does
>>the
>> >>response
>> >> >>> not show up.
>> >> >>> In short:
>> >> >>> wget from VR to www.xyz.com works, response received and saved
>> >> >>> wget from guest to www.xyz.com does not work, network not
>>available
>> >> >>> displayed on guest, response packets not seen on the public
i/f
>>of
>> >>VR at all
>> >> >>> Noel
>> >> >>> 
>> >> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100
>> >> >>>> Subject: Re: Advanced Network - SNAT not working
>> >> >>>> From: msweet.dev@gmail.com
>> >> >>>> To: users@cloudstack.apache.org
>> >> >>>> 
>> >> >>>> Hi Noel,
>> >> >>>> 
>> >> >>>> Can you answer: Does the traffic come back on the public
>> >>interface? and
>> >> >>>> then onto the Guest interface?
>> >> >>>> 
>> >> >>>> Thanks,
>> >> >>>> Marty
>> >> >>>> 
>> >> >>>> 
>> >> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall
>> >><noeldkendall@hotmail.com
>> >> >>>> wrote:
>> >> >>>> 
>> >> >>>>> Indeed, yes, a wget executed on the VR to a public
website
>>works
>> >>just
>> >> >>> fine.
>> >> >>>>> Noel
>> >> >>>>> 
>> >> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100
>> >> >>>>>> Subject: Re: Advanced Network - SNAT not working
>> >> >>>>>> From: msweet.dev@gmail.com
>> >> >>>>>> To: users@cloudstack.apache.org
>> >> >>>>>> 
>> >> >>>>>> Hi Noel,
>> >> >>>>>> 
>> >> >>>>>> Does the traffic come back on the public interface?
and then
>> >>onto the
>> >> >>>>> Guest
>> >> >>>>>> interface?
>> >> >>>>>> 
>> >> >>>>>> Does a wget on the VR work?
>> >> >>>>>> 
>> >> >>>>>> Marty
>> >> >>>>>> 
>> >> >>>>>> 
>> >> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall <
>> >> >>> noeldkendall@hotmail.com
>> >> >>>>>> wrote:
>> >> >>>>>> 
>> >> >>>>>>> I have that Marty. I see the http outbound
request coming in
>>on
>> >>the
>> >> >>>>> guest
>> >> >>>>>>> interface of the VR,and see the http request
being sent out
>>on
>> >>the
>> >> >>>>> public
>> >> >>>>>>> interface of the VR.
>> >> >>>>>>> The traffic is flowing fine from guest to the
outbound i/f of
>> >>the
>> >> >>> VR.
>> >> >>>>>>> This is tcpdump on the public i/f while guest
is doing wget
>>to
>> >> >>>>>>> 6x.xxx.xxx.xxx
>> >> >>>>>>> 
>> >> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00,
>>ethertype
>> >> >>> IPv4
>> >> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 >
6x.xxx.xxx.xx.80:
>> >>Flags
>> >> >>> [S],
>> >> >>>>> seq
>> >> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS
val
>>27489348
>> >>ecr
>> >> >>>>>>> 0,nop,wscale 4], length 0  0x0000:  4500 003c
ad1d 4000 3f06
>> >>2d13
>> >> >>> 0a0b
>> >> >>>>> 4fb2
>> >> >>>>>>>       0x0010:  416e c660 98a2 0050 6ed2 de56
0000 0000
>> >> >>> 0x0020:
>> >> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a   
    0x0030:  01a3
>> >>7444
>> >> >>> 0000
>> >> >>>>>>> 0000 0103 0304
>> >> >>>>>>> 
>> >> >>>>>>> 
>> >> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100
>> >> >>>>>>>> Subject: Re: Advanced Network - SNAT not
working
>> >> >>>>>>>> From: msweet.dev@gmail.com
>> >> >>>>>>>> To: users@cloudstack.apache.org
>> >> >>>>>>>> 
>> >> >>>>>>>> Hi Noel,
>> >> >>>>>>>> 
>> >> >>>>>>>> Can you run a tcpdump on both VR interfaces,
this should
>>make
>> >>it
>> >> >>>>> apparent
>> >> >>>>>>>> what is happening?
>> >> >>>>>>>> 
>> >> >>>>>>>> Thanks,
>> >> >>>>>>>> Marty
>> >> >>>>>>>> 
>> >> >>>>>>>> 
>> >> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall
<
>> >> >>>>> noeldkendall@hotmail.com
>> >> >>>>>>>> wrote:
>> >> >>>>>>>> 
>> >> >>>>>>>>> http://pastebin.com/3FZmFnvZ
>> >> >>>>>>>>> Many thanks Marty.
>> >> >>>>>>>>> Noel
>> >> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55
+0100
>> >> >>>>>>>>>> Subject: Re: Advanced Network -
SNAT not working
>> >> >>>>>>>>>> From: msweet.dev@gmail.com
>> >> >>>>>>>>>> To: users@cloudstack.apache.org
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>> Hi Noel,
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>> Could you put the IP tables on
pastebin? GMail has
>>collapsed
>> >> >>> the
>> >> >>>>>>> lines
>> >> >>>>>>>>>> horrifically.
>> >> >>>>>>>>>> Have you also tried a tcpdump on
both interfaces on the
>>VR?
>> >> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever
it may be called
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>> I would expect worse connectivity
if it was a pure NAT
>>issue,
>> >> >>>>> but I
>> >> >>>>>>> will
>> >> >>>>>>>>>> review the tables later.
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>> Thanks,
>> >> >>>>>>>>>> Marty
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM,
Noel Kendall <
>> >> >>>>>>> noeldkendall@hotmail.com
>> >> >>>>>>>>>> wrote:
>> >> >>>>>>>>>> 
>> >> >>>>>>>>>>> Not seeing return packets on
VR. Suspect, therefore, that
>> >> >>> SNAT
>> >> >>>>> is
>> >> >>>>>>>>> fouled
>> >> >>>>>>>>>>> up in some way.I have been
doing wget to from guest, can
>> >> >>> see
>> >> >>>>> the
>> >> >>>>>>>>> outgoing
>> >> >>>>>>>>>>> request fine, both in the guest
andthe VR.
>> >> >>>>>>>>>>> Could it be that the SNAT table
entries from the
>> >> >>>>> 10.11.0.0/16subnet
>> >> >>>>>>>>> to
>> >> >>>>>>>>>>> dpt www are interfering withthe
SNAT to public ip?? (wild
>> >> >>>>> guess) -
>> >> >>>>>>> not
>> >> >>>>>>>>> an
>> >> >>>>>>>>>>> iptables expert by any stretch
of the imagination
>> >> >>>>>>>>>>> 67.xxx.xxx.56 is the guest
public IP10.11.79.178 is the
>> >> >>> guest
>> >> >>>>> IP on
>> >> >>>>>>>>> guest
>> >> >>>>>>>>>>> network
>> >> >>>>>>>>>>> iptables _L -t nat on the VR
shows...
>> >> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target
    prot opt
>>source
>> >> >>>>>>>>>>> destination         DNAT  
    tcp  --  anywhere
>> >> >>>>>>> anywhere
>> >> >>>>>>>>>>>     tcp dpt:domain to:10.11.0.1
DNAT       tcp  --
>> >> >>> anywhere
>> >> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:www
to:10.11.79.178:80 DNAT
>> >> >>>>>>> tcp  --
>> >> >>>>>>>>>>> anywhere             67.xxx.xxx.56
       tcp dpt:www
>> >> >>>>>>>>> to:10.11.79.178:80DNAT       tcp  --
 anywhere
>> >> >>>>>>> 67.xxx.xxx.56
>> >> >>>>>>>>>       tcp dpt:https
>> >> >>>>>>>>>>> to:10.11.79.178:443 DNAT  
    tcp  --  anywhere
>> >> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:https
>>to:10.11.79.178:443DNAT
>> >> >>>>>>> tcp
>> >> >>>>>>>>> --
>> >> >>>>>>>>>>> anywhere             67.xxx.xxx.56
       tcp dpt:ssh
>> >> >>>>>>>>> to:10.11.79.178:22DNAT       tcp  --
 anywhere
>> >> >>>>>>> 67.xxx.xxx.56
>> >> >>>>>>>>>       tcp dpt:ssh
>> >> >>>>>>>>>>> to:10.11.79.178:22 DNAT   
   tcp  --  anywhere
>> >> >>>>>>>>> 67.xxx.xxx.56
>> >> >>>>>>>>>>>       tcp dpt:ftp to:10.11.79.178:21
DNAT       tcp  --
>> >> >>>>> anywhere
>> >> >>>>>>>>>>>      67.xxx.xxx.56        tcp
dpt:ftp
>>to:10.11.79.178:21DNAT
>> >> >>>>>>>>> tcp
>> >> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56
       tcp
>> >> >>> dpt:5901 to:
>> >> >>>>>>>>>>> 10.11.79.178:5901 DNAT    
  tcp  --  anywhere
>> >> >>>>>>>>> 67.xxx.xxx.56
>> >> >>>>>>>>>>>       tcp dpt:5901 to:10.11.79.178:5901
>> >> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target
    prot opt
>>source
>> >> >>>>>>>>>>> destination         SNAT  
    all  --  anywhere
>> >> >>>>>>> anywhere
>> >> >>>>>>>>>>>     to:67.xxx.xxx.56  SNAT
      all  --  anywhere
>> >> >>>>>>>>> anywhere
>> >> >>>>>>>>>>>         to:67.xxx.xxx.56  SNAT
      all  --  anywhere
>> >> >>>>>>>>>>> anywhere            to:67.xxx.xxx.56
SNAT       all  --
>> >> >>>>> anywhere
>> >> >>>>>>>>>>>  anywhere            to:67.xxx.xxx.56
SNAT       all  --
>> >> >>>>> anywhere
>> >> >>>>>>>>>>>    anywhere            to:67.xxx.xxx.56SNAT
      all  --
>> >> >>>>>>> anywhere
>> >> >>>>>>>>>>>      anywhere            to:67.xxx.xxx.56
SNAT       all
>> >> >>> --
>> >> >>>>>>> anywhere
>> >> >>>>>>>>>>>        anywhere           
to:67.xxx.xxx.56 SNAT
>> >> >>> all  --
>> >> >>>>>>>>> anywhere
>> >> >>>>>>>>>>>          anywhere         
  to:67.xxx.xxx.56 SNAT
>> >> >>> tcp
>> >> >>>>> --
>> >> >>>>>>>>>>> 10.11.0.0/16         myguest
            tcp dpt:www
>> >> >>>>> to:10.11.0.1
>> >> >>>>>>> SNAT
>> >> >>>>>>>>>>>    tcp  --  10.11.0.0/16  
      myguest             tcp
>> >> >>>>>>> dpt:https
>> >> >>>>>>>>>>> to:10.11.0.1 SNAT       tcp
 --  10.11.0.0/16
>> >> >>> myguest
>> >> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT
      tcp  --  10.11.0.0/16
>> >> >>>>>>>>> myguest
>> >> >>>>>>>>>>>            tcp dpt:ftp to:10.11.0.1
SNAT       tcp  --
>> >> >>>>>>> 10.11.0.0/16
>> >> >>>>>>>>>>>    myguest             tcp
dpt:5901 to:10.11.0.1 SNAT
>> >> >>>>> all
>> >> >>>>>>> --
>> >> >>>>>>>>>>> anywhere             anywhere
           to:67.xxx.xxx.56
>> >> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target
    prot opt source
>> >> >>>>>>>>>>> destination         DNAT  
    tcp  --  anywhere
>> >> >>>>>>>>> 67.xxx.xxx.56
>> >> >>>>>>>>>>>      tcp dpt:www to:10.11.79.178:80
DNAT       tcp  --
>> >> >>>>> anywhere
>> >> >>>>>>>>>>>    67.xxx.xxx.56       tcp
dpt:https
>>to:10.11.79.178:443DNAT
>> >> >>>>>>>>> tcp
>> >> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56
      tcp dpt:ssh
>> >> >>> to:
>> >> >>>>>>>>>>> 10.11.79.178:22 DNAT      
tcp  --  anywhere
>> >> >>>>>>> 67.xxx.xxx.56
>> >> >>>>>>>>>>>    tcp dpt:ftp to:10.11.79.178:21
DNAT       tcp  --
>> >> >>>>> anywhere
>> >> >>>>>>>>>>>  67.xxx.xxx.56       tcp dpt:5901
to:10.11.79.178:5901
>> >> >>>>>>>>>>> 
>> >> >>>>>>>>>>>> Date: Sat, 14 Sep 2013
17:25:14 +0100
>> >> >>>>>>>>>>>> Subject: Re: Advanced Network
- SNAT not working
>> >> >>>>>>>>>>>> From: msweet.dev@gmail.com
>> >> >>>>>>>>>>>> To: users@cloudstack.apache.org
>> >> >>>>>>>>>>>> 
>> >> >>>>>>>>>>>> Hi Noel,
>> >> >>>>>>>>>>>> 
>> >> >>>>>>>>>>>> Can you try using telnet
to connect to an external
>> >> >>> webserver?
>> >> >>>>>>> telnet
>> >> >>>>>>>>>>>> www.google.com 80
>> >> >>>>>>>>>>>> Can you also clarify: do
you see the response packets
>> >> >>> reach
>> >> >>>>> the
>> >> >>>>>>> VR
>> >> >>>>>>>>> and/or
>> >> >>>>>>>>>>>> on what interfaces?
>> >> >>>>>>>>>>>> 
>> >> >>>>>>>>>>>> Thanks,
>> >> >>>>>>>>>>>> Marty
>> >> >>>>>>>>>>>> 
>> >> >>>>>>>>>>>> On Saturday, September
14, 2013, Noel Kendall wrote:
>> >> >>>>>>>>>>>> 
>> >> >>>>>>>>>>>>> Guest OS cannot receive
responses to http GETs from
>> >> >>>>> resources
>> >> >>>>>>> on
>> >> >>>>>>>>> the
>> >> >>>>>>>>>>>>> Internet.
>> >> >>>>>>>>>>>>> Network is advanced,
VLAN isolated.
>> >> >>>>>>>>>>>>> What is working:
>> >> >>>>>>>>>>>>> - can browse guest
website from internet- can ssh to
>> >> >>> guest
>> >> >>>>> from
>> >> >>>>>>>>>>> internet-
>> >> >>>>>>>>>>>>> can VPN to guest network
from internet
>> >> >>>>>>>>>>>>> - network VR can access
internet sites no problem
>> >> >>>>>>>>>>>>> What is not working:
>> >> >>>>>>>>>>>>> - guest http traffic
to external website gets to VR on
>> >> >>>>> internal
>> >> >>>>>>>>> NIC,
>> >> >>>>>>>>>>>>> packets forwarded to
external site via external NIC
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Response traffic is
not seen. Appears to be dropped.
>> >> >>>>>>>>>>>>> Have been looking hard
at IPTABLES rules, doing
>> >> >>> tcpdumps,
>> >> >>>>> etc.
>> >> >>>>>>>>>>>>> Am at this point stumped.
>> >> >>>>>>>>>>>>> Any ideas on what could
be wrong, or how to determine
>> >> >>> what
>> >> >>>>>>> could be
>> >> >>>>>>>>>>> wrong?
>> >> >>>>>>>>>>>>> Thanks in advance everyone
who tries to help!
>> >> >>>>>>>>>>>>> N.
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>> 
>> >> >>>>>>>>>>> 
>> >> >>>>>>>>> 
>> >> >>>>>>>>> 
>> >> >>>>>>> 
>> >> >>>>>>> 
>> >> >>>>> 
>> >> >>>>> 
>> >> >>> 
>> >> >>> 
>> >> > 		 	   		  
>> >> 
>> > 		 	   		  
>> 
> 		 	   		  


Mime
View raw message