cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiradeep Vittal <Chiradeep.Vit...@citrix.com>
Subject Re: Advanced Network - SNAT not working
Date Mon, 16 Sep 2013 18:00:29 GMT
Suggest that you stop and start (not reboot) the router from the Admin GUI.

On 9/16/13 5:26 AM, "Noel Kendall" <noeldkendall@hotmail.com> wrote:

>Jayapal, I did a ping test and traced as you suggested. tcpdump
>monitoring was done on the public facing interface of the VR.
>From within the VR, ping to public IP functions correctly, source address
>is the public IP assigned to the VR.
>From within the guest, ping to same public IP, does not function, source
>address is (as you suspected) the IP of guest on the guest network of VR.
>Therefore, it must be: the SNAT rule in iptables in the VR is being
>bypassed... that is, the packets are being forwarded without SNAT being
>performed on them correctly.
>Noel
>
>> From: jayapalreddy.uradi@citrix.com
>> To: users@cloudstack.apache.org
>> Subject: Re: Advanced Network - SNAT not working
>> Date: Mon, 16 Sep 2013 05:14:53 +0000
>> 
>> Hi,
>> 
>> I think when the packets are going out the packets are NATed with
>>private ip, that can't reach back to router.
>> From the VR when you ping public network observe with what source ip
>>address the packet is going out and
>> From the guest VM when you access public n/w observe on VR with what
>>source ip the packet is going out.
>> In later case I think the source ip address is different.
>> 
>> Thanks,
>> Jayapal
>> 
>> 
>> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkendall@hotmail.com>
>>wrote:
>> 
>> > No other NAT. There is nothing but copper between the KVM host
>>machine and the ISP router.There is an L2/L3 switch that the packets
>>travel through. However, there is no forwarding in the switch,just
>>straight through. I've had a well-functioning V4.0.1 environment running
>>on this same configurationin the past. What is new is the conversion to
>>4.1 (which was a clean install).
>> > It's very mysterious, I have never seen anything like this before.
>>There are two other VRs, both having same issue.
>> > I will try your suggestion.
>> > Noel
>> >> Date: Sun, 15 Sep 2013 21:20:41 +0100
>> >> Subject: Re: Advanced Network - SNAT not working
>> >> From: msweet.dev@gmail.com
>> >> To: users@cloudstack.apache.org
>> >> 
>> >> This is mostly confusing that the packets are not seen on the VR
>>public
>> >> interface, seeing as other services are working.
>> >> If it was a local NAT issue then the packet would atleast get into
>>that
>> >> interface. Do you have any upstream devices providing NAT? Or any
>>other VR
>> >> with the issue?
>> >> 
>> >> It may be worth recreating the VR, by stopping and destroying it and
>> >> creating another guest to start a fresh.
>> >> 
>> >> Marty
>> >> 
>> >> 
>> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall
>><noeldkendall@hotmail.com>wrote:
>> >> 
>> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the
guest,
>> >>> while running a tcpdumpon the public i/f of the VR:
>> >>> - I can see the outbound packets going out- I do not see a response
>>packet
>> >>> coming back in
>> >>> FYI there are no firewalls outbound from the KVM host. The host
>>bridges vi
>> >>> CS networkingdirectly out on to the internet via a switch.
>> >>> Note that traffic from outside (ssh, web) can happily traverse the
>>VR to
>> >>> the guest. I get the usualits working html page from the guest.
>>This tells
>> >>> me that there is nothing outbound from the VR thatis filtering
>>packets.
>> >>> Am truly stumped. This is mysterious indeed.
>> >>> From within the VR, can happily telnet to <www.xyz.com> 80 and
>>receive
>> >>> response.Only if packet came from guest and was forwarded does the
>>response
>> >>> not show up.
>> >>> In short:
>> >>> wget from VR to www.xyz.com works, response received and saved
>> >>> wget from guest to www.xyz.com does not work, network not available
>> >>> displayed on guest, response packets not seen on the public i/f of
>>VR at all
>> >>> Noel
>> >>> 
>> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100
>> >>>> Subject: Re: Advanced Network - SNAT not working
>> >>>> From: msweet.dev@gmail.com
>> >>>> To: users@cloudstack.apache.org
>> >>>> 
>> >>>> Hi Noel,
>> >>>> 
>> >>>> Can you answer: Does the traffic come back on the public
>>interface? and
>> >>>> then onto the Guest interface?
>> >>>> 
>> >>>> Thanks,
>> >>>> Marty
>> >>>> 
>> >>>> 
>> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall
>><noeldkendall@hotmail.com
>> >>>> wrote:
>> >>>> 
>> >>>>> Indeed, yes, a wget executed on the VR to a public website works
>>just
>> >>> fine.
>> >>>>> Noel
>> >>>>> 
>> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100
>> >>>>>> Subject: Re: Advanced Network - SNAT not working
>> >>>>>> From: msweet.dev@gmail.com
>> >>>>>> To: users@cloudstack.apache.org
>> >>>>>> 
>> >>>>>> Hi Noel,
>> >>>>>> 
>> >>>>>> Does the traffic come back on the public interface? and
then
>>onto the
>> >>>>> Guest
>> >>>>>> interface?
>> >>>>>> 
>> >>>>>> Does a wget on the VR work?
>> >>>>>> 
>> >>>>>> Marty
>> >>>>>> 
>> >>>>>> 
>> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall <
>> >>> noeldkendall@hotmail.com
>> >>>>>> wrote:
>> >>>>>> 
>> >>>>>>> I have that Marty. I see the http outbound request coming
in on
>>the
>> >>>>> guest
>> >>>>>>> interface of the VR,and see the http request being sent
out on
>>the
>> >>>>> public
>> >>>>>>> interface of the VR.
>> >>>>>>> The traffic is flowing fine from guest to the outbound
i/f of
>>the
>> >>> VR.
>> >>>>>>> This is tcpdump on the public i/f while guest is doing
wget to
>> >>>>>>> 6x.xxx.xxx.xxx
>> >>>>>>> 
>> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00,
ethertype
>> >>> IPv4
>> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80:
>>Flags
>> >>> [S],
>> >>>>> seq
>> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val
27489348
>>ecr
>> >>>>>>> 0,nop,wscale 4], length 0  0x0000:  4500 003c ad1d 4000
3f06
>>2d13
>> >>> 0a0b
>> >>>>> 4fb2
>> >>>>>>>       0x0010:  416e c660 98a2 0050 6ed2 de56 0000 0000
>> >>> 0x0020:
>> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a        0x0030:
 01a3
>>7444
>> >>> 0000
>> >>>>>>> 0000 0103 0304
>> >>>>>>> 
>> >>>>>>> 
>> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100
>> >>>>>>>> Subject: Re: Advanced Network - SNAT not working
>> >>>>>>>> From: msweet.dev@gmail.com
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> 
>> >>>>>>>> Hi Noel,
>> >>>>>>>> 
>> >>>>>>>> Can you run a tcpdump on both VR interfaces, this
should make
>>it
>> >>>>> apparent
>> >>>>>>>> what is happening?
>> >>>>>>>> 
>> >>>>>>>> Thanks,
>> >>>>>>>> Marty
>> >>>>>>>> 
>> >>>>>>>> 
>> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall <
>> >>>>> noeldkendall@hotmail.com
>> >>>>>>>> wrote:
>> >>>>>>>> 
>> >>>>>>>>> http://pastebin.com/3FZmFnvZ
>> >>>>>>>>> Many thanks Marty.
>> >>>>>>>>> Noel
>> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100
>> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not
working
>> >>>>>>>>>> From: msweet.dev@gmail.com
>> >>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>> 
>> >>>>>>>>>> Hi Noel,
>> >>>>>>>>>> 
>> >>>>>>>>>> Could you put the IP tables on pastebin?
GMail has collapsed
>> >>> the
>> >>>>>>> lines
>> >>>>>>>>>> horrifically.
>> >>>>>>>>>> Have you also tried a tcpdump on both interfaces
on the VR?
>> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may
be called
>> >>>>>>>>>> 
>> >>>>>>>>>> I would expect worse connectivity if it
was a pure NAT issue,
>> >>>>> but I
>> >>>>>>> will
>> >>>>>>>>>> review the tables later.
>> >>>>>>>>>> 
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> Marty
>> >>>>>>>>>> 
>> >>>>>>>>>> 
>> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall
<
>> >>>>>>> noeldkendall@hotmail.com
>> >>>>>>>>>> wrote:
>> >>>>>>>>>> 
>> >>>>>>>>>>> Not seeing return packets on VR. Suspect,
therefore, that
>> >>> SNAT
>> >>>>> is
>> >>>>>>>>> fouled
>> >>>>>>>>>>> up in some way.I have been doing wget
to from guest, can
>> >>> see
>> >>>>> the
>> >>>>>>>>> outgoing
>> >>>>>>>>>>> request fine, both in the guest andthe
VR.
>> >>>>>>>>>>> Could it be that the SNAT table entries
from the
>> >>>>> 10.11.0.0/16subnet
>> >>>>>>>>> to
>> >>>>>>>>>>> dpt www are interfering withthe SNAT
to public ip?? (wild
>> >>>>> guess) -
>> >>>>>>> not
>> >>>>>>>>> an
>> >>>>>>>>>>> iptables expert by any stretch of the
imagination
>> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178
is the
>> >>> guest
>> >>>>> IP on
>> >>>>>>>>> guest
>> >>>>>>>>>>> network
>> >>>>>>>>>>> iptables _L -t nat on the VR shows...
>> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target
    prot opt source
>> >>>>>>>>>>> destination         DNAT       tcp 
--  anywhere
>> >>>>>>> anywhere
>> >>>>>>>>>>>     tcp dpt:domain to:10.11.0.1 DNAT
      tcp  --
>> >>> anywhere
>> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:www to:10.11.79.178:80
DNAT
>> >>>>>>> tcp  --
>> >>>>>>>>>>> anywhere             67.xxx.xxx.56 
      tcp dpt:www
>> >>>>>>>>> to:10.11.79.178:80DNAT       tcp  --  anywhere
>> >>>>>>> 67.xxx.xxx.56
>> >>>>>>>>>       tcp dpt:https
>> >>>>>>>>>>> to:10.11.79.178:443 DNAT       tcp 
--  anywhere
>> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:https to:10.11.79.178:443DNAT
>> >>>>>>> tcp
>> >>>>>>>>> --
>> >>>>>>>>>>> anywhere             67.xxx.xxx.56 
      tcp dpt:ssh
>> >>>>>>>>> to:10.11.79.178:22DNAT       tcp  --  anywhere
>> >>>>>>> 67.xxx.xxx.56
>> >>>>>>>>>       tcp dpt:ssh
>> >>>>>>>>>>> to:10.11.79.178:22 DNAT       tcp  --
 anywhere
>> >>>>>>>>> 67.xxx.xxx.56
>> >>>>>>>>>>>       tcp dpt:ftp to:10.11.79.178:21
DNAT       tcp  --
>> >>>>> anywhere
>> >>>>>>>>>>>      67.xxx.xxx.56        tcp dpt:ftp
to:10.11.79.178:21DNAT
>> >>>>>>>>> tcp
>> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56
       tcp
>> >>> dpt:5901 to:
>> >>>>>>>>>>> 10.11.79.178:5901 DNAT       tcp  --
 anywhere
>> >>>>>>>>> 67.xxx.xxx.56
>> >>>>>>>>>>>       tcp dpt:5901 to:10.11.79.178:5901
>> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target
    prot opt source
>> >>>>>>>>>>> destination         SNAT       all 
--  anywhere
>> >>>>>>> anywhere
>> >>>>>>>>>>>     to:67.xxx.xxx.56  SNAT       all
 --  anywhere
>> >>>>>>>>> anywhere
>> >>>>>>>>>>>         to:67.xxx.xxx.56  SNAT     
 all  --  anywhere
>> >>>>>>>>>>> anywhere            to:67.xxx.xxx.56
SNAT       all  --
>> >>>>> anywhere
>> >>>>>>>>>>>  anywhere            to:67.xxx.xxx.56
SNAT       all  --
>> >>>>> anywhere
>> >>>>>>>>>>>    anywhere            to:67.xxx.xxx.56SNAT
      all  --
>> >>>>>>> anywhere
>> >>>>>>>>>>>      anywhere            to:67.xxx.xxx.56
SNAT       all
>> >>> --
>> >>>>>>> anywhere
>> >>>>>>>>>>>        anywhere            to:67.xxx.xxx.56
SNAT
>> >>> all  --
>> >>>>>>>>> anywhere
>> >>>>>>>>>>>          anywhere            to:67.xxx.xxx.56
SNAT
>> >>> tcp
>> >>>>> --
>> >>>>>>>>>>> 10.11.0.0/16         myguest       
     tcp dpt:www
>> >>>>> to:10.11.0.1
>> >>>>>>> SNAT
>> >>>>>>>>>>>    tcp  --  10.11.0.0/16         myguest
            tcp
>> >>>>>>> dpt:https
>> >>>>>>>>>>> to:10.11.0.1 SNAT       tcp  --  10.11.0.0/16
>> >>> myguest
>> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT      
tcp  --  10.11.0.0/16
>> >>>>>>>>> myguest
>> >>>>>>>>>>>            tcp dpt:ftp to:10.11.0.1
SNAT       tcp  --
>> >>>>>>> 10.11.0.0/16
>> >>>>>>>>>>>    myguest             tcp dpt:5901
to:10.11.0.1 SNAT
>> >>>>> all
>> >>>>>>> --
>> >>>>>>>>>>> anywhere             anywhere      
     to:67.xxx.xxx.56
>> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target 
   prot opt source
>> >>>>>>>>>>> destination         DNAT       tcp 
--  anywhere
>> >>>>>>>>> 67.xxx.xxx.56
>> >>>>>>>>>>>      tcp dpt:www to:10.11.79.178:80
DNAT       tcp  --
>> >>>>> anywhere
>> >>>>>>>>>>>    67.xxx.xxx.56       tcp dpt:https
to:10.11.79.178:443DNAT
>> >>>>>>>>> tcp
>> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56
      tcp dpt:ssh
>> >>> to:
>> >>>>>>>>>>> 10.11.79.178:22 DNAT       tcp  -- 
anywhere
>> >>>>>>> 67.xxx.xxx.56
>> >>>>>>>>>>>    tcp dpt:ftp to:10.11.79.178:21 DNAT
      tcp  --
>> >>>>> anywhere
>> >>>>>>>>>>>  67.xxx.xxx.56       tcp dpt:5901 to:10.11.79.178:5901
>> >>>>>>>>>>> 
>> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14
+0100
>> >>>>>>>>>>>> Subject: Re: Advanced Network -
SNAT not working
>> >>>>>>>>>>>> From: msweet.dev@gmail.com
>> >>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>> 
>> >>>>>>>>>>>> Hi Noel,
>> >>>>>>>>>>>> 
>> >>>>>>>>>>>> Can you try using telnet to connect
to an external
>> >>> webserver?
>> >>>>>>> telnet
>> >>>>>>>>>>>> www.google.com 80
>> >>>>>>>>>>>> Can you also clarify: do you see
the response packets
>> >>> reach
>> >>>>> the
>> >>>>>>> VR
>> >>>>>>>>> and/or
>> >>>>>>>>>>>> on what interfaces?
>> >>>>>>>>>>>> 
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Marty
>> >>>>>>>>>>>> 
>> >>>>>>>>>>>> On Saturday, September 14, 2013,
Noel Kendall wrote:
>> >>>>>>>>>>>> 
>> >>>>>>>>>>>>> Guest OS cannot receive responses
to http GETs from
>> >>>>> resources
>> >>>>>>> on
>> >>>>>>>>> the
>> >>>>>>>>>>>>> Internet.
>> >>>>>>>>>>>>> Network is advanced, VLAN isolated.
>> >>>>>>>>>>>>> What is working:
>> >>>>>>>>>>>>> - can browse guest website from
internet- can ssh to
>> >>> guest
>> >>>>> from
>> >>>>>>>>>>> internet-
>> >>>>>>>>>>>>> can VPN to guest network from
internet
>> >>>>>>>>>>>>> - network VR can access internet
sites no problem
>> >>>>>>>>>>>>> What is not working:
>> >>>>>>>>>>>>> - guest http traffic to external
website gets to VR on
>> >>>>> internal
>> >>>>>>>>> NIC,
>> >>>>>>>>>>>>> packets forwarded to external
site via external NIC
>> >>>>>>>>>>>>> 
>> >>>>>>>>>>>>> Response traffic is not seen.
Appears to be dropped.
>> >>>>>>>>>>>>> Have been looking hard at IPTABLES
rules, doing
>> >>> tcpdumps,
>> >>>>> etc.
>> >>>>>>>>>>>>> Am at this point stumped.
>> >>>>>>>>>>>>> Any ideas on what could be wrong,
or how to determine
>> >>> what
>> >>>>>>> could be
>> >>>>>>>>>>> wrong?
>> >>>>>>>>>>>>> Thanks in advance everyone who
tries to help!
>> >>>>>>>>>>>>> N.
>> >>>>>>>>>>>>> 
>> >>>>>>>>>>> 
>> >>>>>>>>>>> 
>> >>>>>>>>> 
>> >>>>>>>>> 
>> >>>>>>> 
>> >>>>>>> 
>> >>>>> 
>> >>>>> 
>> >>> 
>> >>> 
>> > 		 	   		  
>> 
> 		 	   		  


Mime
View raw message