cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noel Kendall <noeldkend...@hotmail.com>
Subject RE: Advanced Network - SNAT not working
Date Mon, 16 Sep 2013 12:26:48 GMT
Jayapal, I did a ping test and traced as you suggested. tcpdump monitoring was done on the
public facing interface of the VR.
>From within the VR, ping to public IP functions correctly, source address is the public
IP assigned to the VR.
>From within the guest, ping to same public IP, does not function, source address is (as
you suspected) the IP of guest on the guest network of VR.
Therefore, it must be: the SNAT rule in iptables in the VR is being bypassed... that is, the
packets are being forwarded without SNAT being performed on them correctly.
Noel

> From: jayapalreddy.uradi@citrix.com
> To: users@cloudstack.apache.org
> Subject: Re: Advanced Network - SNAT not working
> Date: Mon, 16 Sep 2013 05:14:53 +0000
> 
> Hi,
> 
> I think when the packets are going out the packets are NATed with private ip, that can't
reach back to router.
> From the VR when you ping public network observe with what source ip address the packet
is going out and 
> From the guest VM when you access public n/w observe on VR with what source ip the packet
is going out.
> In later case I think the source ip address is different.
> 
> Thanks,
> Jayapal
> 
> 
> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkendall@hotmail.com> wrote:
> 
> > No other NAT. There is nothing but copper between the KVM host machine and the ISP
router.There is an L2/L3 switch that the packets travel through. However, there is no forwarding
in the switch,just straight through. I've had a well-functioning V4.0.1 environment running
on this same configurationin the past. What is new is the conversion to 4.1 (which was a clean
install).
> > It's very mysterious, I have never seen anything like this before. There are two
other VRs, both having same issue.
> > I will try your suggestion.
> > Noel
> >> Date: Sun, 15 Sep 2013 21:20:41 +0100
> >> Subject: Re: Advanced Network - SNAT not working
> >> From: msweet.dev@gmail.com
> >> To: users@cloudstack.apache.org
> >> 
> >> This is mostly confusing that the packets are not seen on the VR public
> >> interface, seeing as other services are working.
> >> If it was a local NAT issue then the packet would atleast get into that
> >> interface. Do you have any upstream devices providing NAT? Or any other VR
> >> with the issue?
> >> 
> >> It may be worth recreating the VR, by stopping and destroying it and
> >> creating another guest to start a fresh.
> >> 
> >> Marty
> >> 
> >> 
> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall <noeldkendall@hotmail.com>wrote:
> >> 
> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the guest,
> >>> while running a tcpdumpon the public i/f of the VR:
> >>> - I can see the outbound packets going out- I do not see a response packet
> >>> coming back in
> >>> FYI there are no firewalls outbound from the KVM host. The host bridges
vi
> >>> CS networkingdirectly out on to the internet via a switch.
> >>> Note that traffic from outside (ssh, web) can happily traverse the VR to
> >>> the guest. I get the usualits working html page from the guest. This tells
> >>> me that there is nothing outbound from the VR thatis filtering packets.
> >>> Am truly stumped. This is mysterious indeed.
> >>> From within the VR, can happily telnet to <www.xyz.com> 80 and receive
> >>> response.Only if packet came from guest and was forwarded does the response
> >>> not show up.
> >>> In short:
> >>> wget from VR to www.xyz.com works, response received and saved
> >>> wget from guest to www.xyz.com does not work, network not available
> >>> displayed on guest, response packets not seen on the public i/f of VR at
all
> >>> Noel
> >>> 
> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100
> >>>> Subject: Re: Advanced Network - SNAT not working
> >>>> From: msweet.dev@gmail.com
> >>>> To: users@cloudstack.apache.org
> >>>> 
> >>>> Hi Noel,
> >>>> 
> >>>> Can you answer: Does the traffic come back on the public interface?
and
> >>>> then onto the Guest interface?
> >>>> 
> >>>> Thanks,
> >>>> Marty
> >>>> 
> >>>> 
> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall <noeldkendall@hotmail.com
> >>>> wrote:
> >>>> 
> >>>>> Indeed, yes, a wget executed on the VR to a public website works
just
> >>> fine.
> >>>>> Noel
> >>>>> 
> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100
> >>>>>> Subject: Re: Advanced Network - SNAT not working
> >>>>>> From: msweet.dev@gmail.com
> >>>>>> To: users@cloudstack.apache.org
> >>>>>> 
> >>>>>> Hi Noel,
> >>>>>> 
> >>>>>> Does the traffic come back on the public interface? and then
onto the
> >>>>> Guest
> >>>>>> interface?
> >>>>>> 
> >>>>>> Does a wget on the VR work?
> >>>>>> 
> >>>>>> Marty
> >>>>>> 
> >>>>>> 
> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall <
> >>> noeldkendall@hotmail.com
> >>>>>> wrote:
> >>>>>> 
> >>>>>>> I have that Marty. I see the http outbound request coming
in on the
> >>>>> guest
> >>>>>>> interface of the VR,and see the http request being sent
out on the
> >>>>> public
> >>>>>>> interface of the VR.
> >>>>>>> The traffic is flowing fine from guest to the outbound i/f
of the
> >>> VR.
> >>>>>>> This is tcpdump on the public i/f while guest is doing wget
to
> >>>>>>> 6x.xxx.xxx.xxx
> >>>>>>> 
> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00,
ethertype
> >>> IPv4
> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80:
Flags
> >>> [S],
> >>>>> seq
> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348
ecr
> >>>>>>> 0,nop,wscale 4], length 0  0x0000:  4500 003c ad1d 4000
3f06 2d13
> >>> 0a0b
> >>>>> 4fb2
> >>>>>>>       0x0010:  416e c660 98a2 0050 6ed2 de56 0000 0000
> >>> 0x0020:
> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a        0x0030: 
01a3 7444
> >>> 0000
> >>>>>>> 0000 0103 0304
> >>>>>>> 
> >>>>>>> 
> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100
> >>>>>>>> Subject: Re: Advanced Network - SNAT not working
> >>>>>>>> From: msweet.dev@gmail.com
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> 
> >>>>>>>> Hi Noel,
> >>>>>>>> 
> >>>>>>>> Can you run a tcpdump on both VR interfaces, this should
make it
> >>>>> apparent
> >>>>>>>> what is happening?
> >>>>>>>> 
> >>>>>>>> Thanks,
> >>>>>>>> Marty
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall <
> >>>>> noeldkendall@hotmail.com
> >>>>>>>> wrote:
> >>>>>>>> 
> >>>>>>>>> http://pastebin.com/3FZmFnvZ
> >>>>>>>>> Many thanks Marty.
> >>>>>>>>> Noel
> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100
> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working
> >>>>>>>>>> From: msweet.dev@gmail.com
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> 
> >>>>>>>>>> Hi Noel,
> >>>>>>>>>> 
> >>>>>>>>>> Could you put the IP tables on pastebin? GMail
has collapsed
> >>> the
> >>>>>>> lines
> >>>>>>>>>> horrifically.
> >>>>>>>>>> Have you also tried a tcpdump on both interfaces
on the VR?
> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be
called
> >>>>>>>>>> 
> >>>>>>>>>> I would expect worse connectivity if it was
a pure NAT issue,
> >>>>> but I
> >>>>>>> will
> >>>>>>>>>> review the tables later.
> >>>>>>>>>> 
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Marty
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall
<
> >>>>>>> noeldkendall@hotmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>> 
> >>>>>>>>>>> Not seeing return packets on VR. Suspect,
therefore, that
> >>> SNAT
> >>>>> is
> >>>>>>>>> fouled
> >>>>>>>>>>> up in some way.I have been doing wget to
from guest, can
> >>> see
> >>>>> the
> >>>>>>>>> outgoing
> >>>>>>>>>>> request fine, both in the guest andthe VR.
> >>>>>>>>>>> Could it be that the SNAT table entries
from the
> >>>>> 10.11.0.0/16subnet
> >>>>>>>>> to
> >>>>>>>>>>> dpt www are interfering withthe SNAT to
public ip?? (wild
> >>>>> guess) -
> >>>>>>> not
> >>>>>>>>> an
> >>>>>>>>>>> iptables expert by any stretch of the imagination
> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178
is the
> >>> guest
> >>>>> IP on
> >>>>>>>>> guest
> >>>>>>>>>>> network
> >>>>>>>>>>> iptables _L -t nat on the VR shows...
> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target 
   prot opt source
> >>>>>>>>>>> destination         DNAT       tcp  -- 
anywhere
> >>>>>>> anywhere
> >>>>>>>>>>>     tcp dpt:domain to:10.11.0.1 DNAT   
   tcp  --
> >>> anywhere
> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:www to:10.11.79.178:80
DNAT
> >>>>>>> tcp  --
> >>>>>>>>>>> anywhere             67.xxx.xxx.56     
  tcp dpt:www
> >>>>>>>>> to:10.11.79.178:80DNAT       tcp  --  anywhere
> >>>>>>> 67.xxx.xxx.56
> >>>>>>>>>       tcp dpt:https
> >>>>>>>>>>> to:10.11.79.178:443 DNAT       tcp  -- 
anywhere
> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:https to:10.11.79.178:443DNAT
> >>>>>>> tcp
> >>>>>>>>> --
> >>>>>>>>>>> anywhere             67.xxx.xxx.56     
  tcp dpt:ssh
> >>>>>>>>> to:10.11.79.178:22DNAT       tcp  --  anywhere
> >>>>>>> 67.xxx.xxx.56
> >>>>>>>>>       tcp dpt:ssh
> >>>>>>>>>>> to:10.11.79.178:22 DNAT       tcp  --  anywhere
> >>>>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>       tcp dpt:ftp to:10.11.79.178:21 DNAT
      tcp  --
> >>>>> anywhere
> >>>>>>>>>>>      67.xxx.xxx.56        tcp dpt:ftp to:10.11.79.178:21DNAT
> >>>>>>>>> tcp
> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56 
      tcp
> >>> dpt:5901 to:
> >>>>>>>>>>> 10.11.79.178:5901 DNAT       tcp  --  anywhere
> >>>>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>       tcp dpt:5901 to:10.11.79.178:5901
> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target
    prot opt source
> >>>>>>>>>>> destination         SNAT       all  -- 
anywhere
> >>>>>>> anywhere
> >>>>>>>>>>>     to:67.xxx.xxx.56  SNAT       all  --
 anywhere
> >>>>>>>>> anywhere
> >>>>>>>>>>>         to:67.xxx.xxx.56  SNAT       all
 --  anywhere
> >>>>>>>>>>> anywhere            to:67.xxx.xxx.56 SNAT
      all  --
> >>>>> anywhere
> >>>>>>>>>>>  anywhere            to:67.xxx.xxx.56 SNAT
      all  --
> >>>>> anywhere
> >>>>>>>>>>>    anywhere            to:67.xxx.xxx.56SNAT
      all  --
> >>>>>>> anywhere
> >>>>>>>>>>>      anywhere            to:67.xxx.xxx.56
SNAT       all
> >>> --
> >>>>>>> anywhere
> >>>>>>>>>>>        anywhere            to:67.xxx.xxx.56
SNAT
> >>> all  --
> >>>>>>>>> anywhere
> >>>>>>>>>>>          anywhere            to:67.xxx.xxx.56
SNAT
> >>> tcp
> >>>>> --
> >>>>>>>>>>> 10.11.0.0/16         myguest           
 tcp dpt:www
> >>>>> to:10.11.0.1
> >>>>>>> SNAT
> >>>>>>>>>>>    tcp  --  10.11.0.0/16         myguest
            tcp
> >>>>>>> dpt:https
> >>>>>>>>>>> to:10.11.0.1 SNAT       tcp  --  10.11.0.0/16
> >>> myguest
> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT       tcp
 --  10.11.0.0/16
> >>>>>>>>> myguest
> >>>>>>>>>>>            tcp dpt:ftp to:10.11.0.1 SNAT
      tcp  --
> >>>>>>> 10.11.0.0/16
> >>>>>>>>>>>    myguest             tcp dpt:5901 to:10.11.0.1
SNAT
> >>>>> all
> >>>>>>> --
> >>>>>>>>>>> anywhere             anywhere          
 to:67.xxx.xxx.56
> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target     prot
opt source
> >>>>>>>>>>> destination         DNAT       tcp  -- 
anywhere
> >>>>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>      tcp dpt:www to:10.11.79.178:80 DNAT
      tcp  --
> >>>>> anywhere
> >>>>>>>>>>>    67.xxx.xxx.56       tcp dpt:https to:10.11.79.178:443DNAT
> >>>>>>>>> tcp
> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56 
     tcp dpt:ssh
> >>> to:
> >>>>>>>>>>> 10.11.79.178:22 DNAT       tcp  --  anywhere
> >>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>    tcp dpt:ftp to:10.11.79.178:21 DNAT 
     tcp  --
> >>>>> anywhere
> >>>>>>>>>>>  67.xxx.xxx.56       tcp dpt:5901 to:10.11.79.178:5901
> >>>>>>>>>>> 
> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100
> >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT
not working
> >>>>>>>>>>>> From: msweet.dev@gmail.com
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Hi Noel,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Can you try using telnet to connect
to an external
> >>> webserver?
> >>>>>>> telnet
> >>>>>>>>>>>> www.google.com 80
> >>>>>>>>>>>> Can you also clarify: do you see the
response packets
> >>> reach
> >>>>> the
> >>>>>>> VR
> >>>>>>>>> and/or
> >>>>>>>>>>>> on what interfaces?
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Marty
> >>>>>>>>>>>> 
> >>>>>>>>>>>> On Saturday, September 14, 2013, Noel
Kendall wrote:
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> Guest OS cannot receive responses
to http GETs from
> >>>>> resources
> >>>>>>> on
> >>>>>>>>> the
> >>>>>>>>>>>>> Internet.
> >>>>>>>>>>>>> Network is advanced, VLAN isolated.
> >>>>>>>>>>>>> What is working:
> >>>>>>>>>>>>> - can browse guest website from
internet- can ssh to
> >>> guest
> >>>>> from
> >>>>>>>>>>> internet-
> >>>>>>>>>>>>> can VPN to guest network from internet
> >>>>>>>>>>>>> - network VR can access internet
sites no problem
> >>>>>>>>>>>>> What is not working:
> >>>>>>>>>>>>> - guest http traffic to external
website gets to VR on
> >>>>> internal
> >>>>>>>>> NIC,
> >>>>>>>>>>>>> packets forwarded to external site
via external NIC
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Response traffic is not seen. Appears
to be dropped.
> >>>>>>>>>>>>> Have been looking hard at IPTABLES
rules, doing
> >>> tcpdumps,
> >>>>> etc.
> >>>>>>>>>>>>> Am at this point stumped.
> >>>>>>>>>>>>> Any ideas on what could be wrong,
or how to determine
> >>> what
> >>>>>>> could be
> >>>>>>>>>>> wrong?
> >>>>>>>>>>>>> Thanks in advance everyone who tries
to help!
> >>>>>>>>>>>>> N.
> >>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>> 
> >>>>> 
> >>> 
> >>> 
> > 		 	   		  
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message