cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noel Kendall <noeldkend...@hotmail.com>
Subject RE: Advanced Network - SNAT not working
Date Mon, 16 Sep 2013 19:04:02 GMT
That has worked Chiradeep. What could have caused this problem? Is it somethingthat should
be fixed?
Thanks for the simple and rather effective suggestion!
Noel

> From: Chiradeep.Vittal@citrix.com
> To: users@cloudstack.apache.org
> Subject: Re: Advanced Network - SNAT not working
> Date: Mon, 16 Sep 2013 18:00:29 +0000
> 
> Suggest that you stop and start (not reboot) the router from the Admin GUI.
> 
> On 9/16/13 5:26 AM, "Noel Kendall" <noeldkendall@hotmail.com> wrote:
> 
> >Jayapal, I did a ping test and traced as you suggested. tcpdump
> >monitoring was done on the public facing interface of the VR.
> >From within the VR, ping to public IP functions correctly, source address
> >is the public IP assigned to the VR.
> >From within the guest, ping to same public IP, does not function, source
> >address is (as you suspected) the IP of guest on the guest network of VR.
> >Therefore, it must be: the SNAT rule in iptables in the VR is being
> >bypassed... that is, the packets are being forwarded without SNAT being
> >performed on them correctly.
> >Noel
> >
> >> From: jayapalreddy.uradi@citrix.com
> >> To: users@cloudstack.apache.org
> >> Subject: Re: Advanced Network - SNAT not working
> >> Date: Mon, 16 Sep 2013 05:14:53 +0000
> >> 
> >> Hi,
> >> 
> >> I think when the packets are going out the packets are NATed with
> >>private ip, that can't reach back to router.
> >> From the VR when you ping public network observe with what source ip
> >>address the packet is going out and
> >> From the guest VM when you access public n/w observe on VR with what
> >>source ip the packet is going out.
> >> In later case I think the source ip address is different.
> >> 
> >> Thanks,
> >> Jayapal
> >> 
> >> 
> >> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkendall@hotmail.com>
> >>wrote:
> >> 
> >> > No other NAT. There is nothing but copper between the KVM host
> >>machine and the ISP router.There is an L2/L3 switch that the packets
> >>travel through. However, there is no forwarding in the switch,just
> >>straight through. I've had a well-functioning V4.0.1 environment running
> >>on this same configurationin the past. What is new is the conversion to
> >>4.1 (which was a clean install).
> >> > It's very mysterious, I have never seen anything like this before.
> >>There are two other VRs, both having same issue.
> >> > I will try your suggestion.
> >> > Noel
> >> >> Date: Sun, 15 Sep 2013 21:20:41 +0100
> >> >> Subject: Re: Advanced Network - SNAT not working
> >> >> From: msweet.dev@gmail.com
> >> >> To: users@cloudstack.apache.org
> >> >> 
> >> >> This is mostly confusing that the packets are not seen on the VR
> >>public
> >> >> interface, seeing as other services are working.
> >> >> If it was a local NAT issue then the packet would atleast get into
> >>that
> >> >> interface. Do you have any upstream devices providing NAT? Or any
> >>other VR
> >> >> with the issue?
> >> >> 
> >> >> It may be worth recreating the VR, by stopping and destroying it and
> >> >> creating another guest to start a fresh.
> >> >> 
> >> >> Marty
> >> >> 
> >> >> 
> >> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall
> >><noeldkendall@hotmail.com>wrote:
> >> >> 
> >> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in
the guest,
> >> >>> while running a tcpdumpon the public i/f of the VR:
> >> >>> - I can see the outbound packets going out- I do not see a response
> >>packet
> >> >>> coming back in
> >> >>> FYI there are no firewalls outbound from the KVM host. The host
> >>bridges vi
> >> >>> CS networkingdirectly out on to the internet via a switch.
> >> >>> Note that traffic from outside (ssh, web) can happily traverse
the
> >>VR to
> >> >>> the guest. I get the usualits working html page from the guest.
> >>This tells
> >> >>> me that there is nothing outbound from the VR thatis filtering
> >>packets.
> >> >>> Am truly stumped. This is mysterious indeed.
> >> >>> From within the VR, can happily telnet to <www.xyz.com> 80
and
> >>receive
> >> >>> response.Only if packet came from guest and was forwarded does
the
> >>response
> >> >>> not show up.
> >> >>> In short:
> >> >>> wget from VR to www.xyz.com works, response received and saved
> >> >>> wget from guest to www.xyz.com does not work, network not available
> >> >>> displayed on guest, response packets not seen on the public i/f
of
> >>VR at all
> >> >>> Noel
> >> >>> 
> >> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100
> >> >>>> Subject: Re: Advanced Network - SNAT not working
> >> >>>> From: msweet.dev@gmail.com
> >> >>>> To: users@cloudstack.apache.org
> >> >>>> 
> >> >>>> Hi Noel,
> >> >>>> 
> >> >>>> Can you answer: Does the traffic come back on the public
> >>interface? and
> >> >>>> then onto the Guest interface?
> >> >>>> 
> >> >>>> Thanks,
> >> >>>> Marty
> >> >>>> 
> >> >>>> 
> >> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall
> >><noeldkendall@hotmail.com
> >> >>>> wrote:
> >> >>>> 
> >> >>>>> Indeed, yes, a wget executed on the VR to a public website
works
> >>just
> >> >>> fine.
> >> >>>>> Noel
> >> >>>>> 
> >> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100
> >> >>>>>> Subject: Re: Advanced Network - SNAT not working
> >> >>>>>> From: msweet.dev@gmail.com
> >> >>>>>> To: users@cloudstack.apache.org
> >> >>>>>> 
> >> >>>>>> Hi Noel,
> >> >>>>>> 
> >> >>>>>> Does the traffic come back on the public interface?
and then
> >>onto the
> >> >>>>> Guest
> >> >>>>>> interface?
> >> >>>>>> 
> >> >>>>>> Does a wget on the VR work?
> >> >>>>>> 
> >> >>>>>> Marty
> >> >>>>>> 
> >> >>>>>> 
> >> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall <
> >> >>> noeldkendall@hotmail.com
> >> >>>>>> wrote:
> >> >>>>>> 
> >> >>>>>>> I have that Marty. I see the http outbound request
coming in on
> >>the
> >> >>>>> guest
> >> >>>>>>> interface of the VR,and see the http request being
sent out on
> >>the
> >> >>>>> public
> >> >>>>>>> interface of the VR.
> >> >>>>>>> The traffic is flowing fine from guest to the outbound
i/f of
> >>the
> >> >>> VR.
> >> >>>>>>> This is tcpdump on the public i/f while guest is
doing wget to
> >> >>>>>>> 6x.xxx.xxx.xxx
> >> >>>>>>> 
> >> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00,
ethertype
> >> >>> IPv4
> >> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80:
> >>Flags
> >> >>> [S],
> >> >>>>> seq
> >> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS
val 27489348
> >>ecr
> >> >>>>>>> 0,nop,wscale 4], length 0  0x0000:  4500 003c ad1d
4000 3f06
> >>2d13
> >> >>> 0a0b
> >> >>>>> 4fb2
> >> >>>>>>>       0x0010:  416e c660 98a2 0050 6ed2 de56 0000
0000
> >> >>> 0x0020:
> >> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a       
0x0030:  01a3
> >>7444
> >> >>> 0000
> >> >>>>>>> 0000 0103 0304
> >> >>>>>>> 
> >> >>>>>>> 
> >> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100
> >> >>>>>>>> Subject: Re: Advanced Network - SNAT not working
> >> >>>>>>>> From: msweet.dev@gmail.com
> >> >>>>>>>> To: users@cloudstack.apache.org
> >> >>>>>>>> 
> >> >>>>>>>> Hi Noel,
> >> >>>>>>>> 
> >> >>>>>>>> Can you run a tcpdump on both VR interfaces,
this should make
> >>it
> >> >>>>> apparent
> >> >>>>>>>> what is happening?
> >> >>>>>>>> 
> >> >>>>>>>> Thanks,
> >> >>>>>>>> Marty
> >> >>>>>>>> 
> >> >>>>>>>> 
> >> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall
<
> >> >>>>> noeldkendall@hotmail.com
> >> >>>>>>>> wrote:
> >> >>>>>>>> 
> >> >>>>>>>>> http://pastebin.com/3FZmFnvZ
> >> >>>>>>>>> Many thanks Marty.
> >> >>>>>>>>> Noel
> >> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100
> >> >>>>>>>>>> Subject: Re: Advanced Network - SNAT
not working
> >> >>>>>>>>>> From: msweet.dev@gmail.com
> >> >>>>>>>>>> To: users@cloudstack.apache.org
> >> >>>>>>>>>> 
> >> >>>>>>>>>> Hi Noel,
> >> >>>>>>>>>> 
> >> >>>>>>>>>> Could you put the IP tables on pastebin?
GMail has collapsed
> >> >>> the
> >> >>>>>>> lines
> >> >>>>>>>>>> horrifically.
> >> >>>>>>>>>> Have you also tried a tcpdump on both
interfaces on the VR?
> >> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever
it may be called
> >> >>>>>>>>>> 
> >> >>>>>>>>>> I would expect worse connectivity if
it was a pure NAT issue,
> >> >>>>> but I
> >> >>>>>>> will
> >> >>>>>>>>>> review the tables later.
> >> >>>>>>>>>> 
> >> >>>>>>>>>> Thanks,
> >> >>>>>>>>>> Marty
> >> >>>>>>>>>> 
> >> >>>>>>>>>> 
> >> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel
Kendall <
> >> >>>>>>> noeldkendall@hotmail.com
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>> 
> >> >>>>>>>>>>> Not seeing return packets on VR.
Suspect, therefore, that
> >> >>> SNAT
> >> >>>>> is
> >> >>>>>>>>> fouled
> >> >>>>>>>>>>> up in some way.I have been doing
wget to from guest, can
> >> >>> see
> >> >>>>> the
> >> >>>>>>>>> outgoing
> >> >>>>>>>>>>> request fine, both in the guest
andthe VR.
> >> >>>>>>>>>>> Could it be that the SNAT table
entries from the
> >> >>>>> 10.11.0.0/16subnet
> >> >>>>>>>>> to
> >> >>>>>>>>>>> dpt www are interfering withthe
SNAT to public ip?? (wild
> >> >>>>> guess) -
> >> >>>>>>> not
> >> >>>>>>>>> an
> >> >>>>>>>>>>> iptables expert by any stretch
of the imagination
> >> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public
IP10.11.79.178 is the
> >> >>> guest
> >> >>>>> IP on
> >> >>>>>>>>> guest
> >> >>>>>>>>>>> network
> >> >>>>>>>>>>> iptables _L -t nat on the VR shows...
> >> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target
    prot opt source
> >> >>>>>>>>>>> destination         DNAT      
tcp  --  anywhere
> >> >>>>>>> anywhere
> >> >>>>>>>>>>>     tcp dpt:domain to:10.11.0.1
DNAT       tcp  --
> >> >>> anywhere
> >> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:www
to:10.11.79.178:80 DNAT
> >> >>>>>>> tcp  --
> >> >>>>>>>>>>> anywhere             67.xxx.xxx.56
       tcp dpt:www
> >> >>>>>>>>> to:10.11.79.178:80DNAT       tcp  --  anywhere
> >> >>>>>>> 67.xxx.xxx.56
> >> >>>>>>>>>       tcp dpt:https
> >> >>>>>>>>>>> to:10.11.79.178:443 DNAT      
tcp  --  anywhere
> >> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:https
to:10.11.79.178:443DNAT
> >> >>>>>>> tcp
> >> >>>>>>>>> --
> >> >>>>>>>>>>> anywhere             67.xxx.xxx.56
       tcp dpt:ssh
> >> >>>>>>>>> to:10.11.79.178:22DNAT       tcp  --  anywhere
> >> >>>>>>> 67.xxx.xxx.56
> >> >>>>>>>>>       tcp dpt:ssh
> >> >>>>>>>>>>> to:10.11.79.178:22 DNAT       tcp
 --  anywhere
> >> >>>>>>>>> 67.xxx.xxx.56
> >> >>>>>>>>>>>       tcp dpt:ftp to:10.11.79.178:21
DNAT       tcp  --
> >> >>>>> anywhere
> >> >>>>>>>>>>>      67.xxx.xxx.56        tcp dpt:ftp
to:10.11.79.178:21DNAT
> >> >>>>>>>>> tcp
> >> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56
       tcp
> >> >>> dpt:5901 to:
> >> >>>>>>>>>>> 10.11.79.178:5901 DNAT       tcp
 --  anywhere
> >> >>>>>>>>> 67.xxx.xxx.56
> >> >>>>>>>>>>>       tcp dpt:5901 to:10.11.79.178:5901
> >> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target
    prot opt source
> >> >>>>>>>>>>> destination         SNAT      
all  --  anywhere
> >> >>>>>>> anywhere
> >> >>>>>>>>>>>     to:67.xxx.xxx.56  SNAT    
  all  --  anywhere
> >> >>>>>>>>> anywhere
> >> >>>>>>>>>>>         to:67.xxx.xxx.56  SNAT
      all  --  anywhere
> >> >>>>>>>>>>> anywhere            to:67.xxx.xxx.56
SNAT       all  --
> >> >>>>> anywhere
> >> >>>>>>>>>>>  anywhere            to:67.xxx.xxx.56
SNAT       all  --
> >> >>>>> anywhere
> >> >>>>>>>>>>>    anywhere            to:67.xxx.xxx.56SNAT
      all  --
> >> >>>>>>> anywhere
> >> >>>>>>>>>>>      anywhere            to:67.xxx.xxx.56
SNAT       all
> >> >>> --
> >> >>>>>>> anywhere
> >> >>>>>>>>>>>        anywhere            to:67.xxx.xxx.56
SNAT
> >> >>> all  --
> >> >>>>>>>>> anywhere
> >> >>>>>>>>>>>          anywhere            to:67.xxx.xxx.56
SNAT
> >> >>> tcp
> >> >>>>> --
> >> >>>>>>>>>>> 10.11.0.0/16         myguest  
          tcp dpt:www
> >> >>>>> to:10.11.0.1
> >> >>>>>>> SNAT
> >> >>>>>>>>>>>    tcp  --  10.11.0.0/16      
  myguest             tcp
> >> >>>>>>> dpt:https
> >> >>>>>>>>>>> to:10.11.0.1 SNAT       tcp  --
 10.11.0.0/16
> >> >>> myguest
> >> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT 
     tcp  --  10.11.0.0/16
> >> >>>>>>>>> myguest
> >> >>>>>>>>>>>            tcp dpt:ftp to:10.11.0.1
SNAT       tcp  --
> >> >>>>>>> 10.11.0.0/16
> >> >>>>>>>>>>>    myguest             tcp dpt:5901
to:10.11.0.1 SNAT
> >> >>>>> all
> >> >>>>>>> --
> >> >>>>>>>>>>> anywhere             anywhere 
          to:67.xxx.xxx.56
> >> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target
    prot opt source
> >> >>>>>>>>>>> destination         DNAT      
tcp  --  anywhere
> >> >>>>>>>>> 67.xxx.xxx.56
> >> >>>>>>>>>>>      tcp dpt:www to:10.11.79.178:80
DNAT       tcp  --
> >> >>>>> anywhere
> >> >>>>>>>>>>>    67.xxx.xxx.56       tcp dpt:https
to:10.11.79.178:443DNAT
> >> >>>>>>>>> tcp
> >> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56
      tcp dpt:ssh
> >> >>> to:
> >> >>>>>>>>>>> 10.11.79.178:22 DNAT       tcp
 --  anywhere
> >> >>>>>>> 67.xxx.xxx.56
> >> >>>>>>>>>>>    tcp dpt:ftp to:10.11.79.178:21
DNAT       tcp  --
> >> >>>>> anywhere
> >> >>>>>>>>>>>  67.xxx.xxx.56       tcp dpt:5901
to:10.11.79.178:5901
> >> >>>>>>>>>>> 
> >> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14
+0100
> >> >>>>>>>>>>>> Subject: Re: Advanced Network
- SNAT not working
> >> >>>>>>>>>>>> From: msweet.dev@gmail.com
> >> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >> >>>>>>>>>>>> 
> >> >>>>>>>>>>>> Hi Noel,
> >> >>>>>>>>>>>> 
> >> >>>>>>>>>>>> Can you try using telnet to
connect to an external
> >> >>> webserver?
> >> >>>>>>> telnet
> >> >>>>>>>>>>>> www.google.com 80
> >> >>>>>>>>>>>> Can you also clarify: do you
see the response packets
> >> >>> reach
> >> >>>>> the
> >> >>>>>>> VR
> >> >>>>>>>>> and/or
> >> >>>>>>>>>>>> on what interfaces?
> >> >>>>>>>>>>>> 
> >> >>>>>>>>>>>> Thanks,
> >> >>>>>>>>>>>> Marty
> >> >>>>>>>>>>>> 
> >> >>>>>>>>>>>> On Saturday, September 14,
2013, Noel Kendall wrote:
> >> >>>>>>>>>>>> 
> >> >>>>>>>>>>>>> Guest OS cannot receive
responses to http GETs from
> >> >>>>> resources
> >> >>>>>>> on
> >> >>>>>>>>> the
> >> >>>>>>>>>>>>> Internet.
> >> >>>>>>>>>>>>> Network is advanced, VLAN
isolated.
> >> >>>>>>>>>>>>> What is working:
> >> >>>>>>>>>>>>> - can browse guest website
from internet- can ssh to
> >> >>> guest
> >> >>>>> from
> >> >>>>>>>>>>> internet-
> >> >>>>>>>>>>>>> can VPN to guest network
from internet
> >> >>>>>>>>>>>>> - network VR can access
internet sites no problem
> >> >>>>>>>>>>>>> What is not working:
> >> >>>>>>>>>>>>> - guest http traffic to
external website gets to VR on
> >> >>>>> internal
> >> >>>>>>>>> NIC,
> >> >>>>>>>>>>>>> packets forwarded to external
site via external NIC
> >> >>>>>>>>>>>>> 
> >> >>>>>>>>>>>>> Response traffic is not
seen. Appears to be dropped.
> >> >>>>>>>>>>>>> Have been looking hard
at IPTABLES rules, doing
> >> >>> tcpdumps,
> >> >>>>> etc.
> >> >>>>>>>>>>>>> Am at this point stumped.
> >> >>>>>>>>>>>>> Any ideas on what could
be wrong, or how to determine
> >> >>> what
> >> >>>>>>> could be
> >> >>>>>>>>>>> wrong?
> >> >>>>>>>>>>>>> Thanks in advance everyone
who tries to help!
> >> >>>>>>>>>>>>> N.
> >> >>>>>>>>>>>>> 
> >> >>>>>>>>>>> 
> >> >>>>>>>>>>> 
> >> >>>>>>>>> 
> >> >>>>>>>>> 
> >> >>>>>>> 
> >> >>>>>>> 
> >> >>>>> 
> >> >>>>> 
> >> >>> 
> >> >>> 
> >> > 		 	   		  
> >> 
> > 		 	   		  
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message