cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noel Kendall <noeldkend...@hotmail.com>
Subject RE: Advanced Network - SNAT not working
Date Mon, 16 Sep 2013 12:16:58 GMT
I will have a look at that Jayapal - it's a good suggestion and useful bit of data. Doesn't
the SNAT rule on created on the VR take care of this though?Noel

> From: jayapalreddy.uradi@citrix.com
> To: users@cloudstack.apache.org
> Subject: Re: Advanced Network - SNAT not working
> Date: Mon, 16 Sep 2013 05:14:53 +0000
> 
> Hi,
> 
> I think when the packets are going out the packets are NATed with private ip, that can't
reach back to router.
> From the VR when you ping public network observe with what source ip address the packet
is going out and 
> From the guest VM when you access public n/w observe on VR with what source ip the packet
is going out.
> In later case I think the source ip address is different.
> 
> Thanks,
> Jayapal
> 
> 
> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkendall@hotmail.com> wrote:
> 
> > No other NAT. There is nothing but copper between the KVM host machine and the ISP
router.There is an L2/L3 switch that the packets travel through. However, there is no forwarding
in the switch,just straight through. I've had a well-functioning V4.0.1 environment running
on this same configurationin the past. What is new is the conversion to 4.1 (which was a clean
install).
> > It's very mysterious, I have never seen anything like this before. There are two
other VRs, both having same issue.
> > I will try your suggestion.
> > Noel
> >> Date: Sun, 15 Sep 2013 21:20:41 +0100
> >> Subject: Re: Advanced Network - SNAT not working
> >> From: msweet.dev@gmail.com
> >> To: users@cloudstack.apache.org
> >> 
> >> This is mostly confusing that the packets are not seen on the VR public
> >> interface, seeing as other services are working.
> >> If it was a local NAT issue then the packet would atleast get into that
> >> interface. Do you have any upstream devices providing NAT? Or any other VR
> >> with the issue?
> >> 
> >> It may be worth recreating the VR, by stopping and destroying it and
> >> creating another guest to start a fresh.
> >> 
> >> Marty
> >> 
> >> 
> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall <noeldkendall@hotmail.com>wrote:
> >> 
> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the guest,
> >>> while running a tcpdumpon the public i/f of the VR:
> >>> - I can see the outbound packets going out- I do not see a response packet
> >>> coming back in
> >>> FYI there are no firewalls outbound from the KVM host. The host bridges
vi
> >>> CS networkingdirectly out on to the internet via a switch.
> >>> Note that traffic from outside (ssh, web) can happily traverse the VR to
> >>> the guest. I get the usualits working html page from the guest. This tells
> >>> me that there is nothing outbound from the VR thatis filtering packets.
> >>> Am truly stumped. This is mysterious indeed.
> >>> From within the VR, can happily telnet to <www.xyz.com> 80 and receive
> >>> response.Only if packet came from guest and was forwarded does the response
> >>> not show up.
> >>> In short:
> >>> wget from VR to www.xyz.com works, response received and saved
> >>> wget from guest to www.xyz.com does not work, network not available
> >>> displayed on guest, response packets not seen on the public i/f of VR at
all
> >>> Noel
> >>> 
> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100
> >>>> Subject: Re: Advanced Network - SNAT not working
> >>>> From: msweet.dev@gmail.com
> >>>> To: users@cloudstack.apache.org
> >>>> 
> >>>> Hi Noel,
> >>>> 
> >>>> Can you answer: Does the traffic come back on the public interface?
and
> >>>> then onto the Guest interface?
> >>>> 
> >>>> Thanks,
> >>>> Marty
> >>>> 
> >>>> 
> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall <noeldkendall@hotmail.com
> >>>> wrote:
> >>>> 
> >>>>> Indeed, yes, a wget executed on the VR to a public website works
just
> >>> fine.
> >>>>> Noel
> >>>>> 
> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100
> >>>>>> Subject: Re: Advanced Network - SNAT not working
> >>>>>> From: msweet.dev@gmail.com
> >>>>>> To: users@cloudstack.apache.org
> >>>>>> 
> >>>>>> Hi Noel,
> >>>>>> 
> >>>>>> Does the traffic come back on the public interface? and then
onto the
> >>>>> Guest
> >>>>>> interface?
> >>>>>> 
> >>>>>> Does a wget on the VR work?
> >>>>>> 
> >>>>>> Marty
> >>>>>> 
> >>>>>> 
> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall <
> >>> noeldkendall@hotmail.com
> >>>>>> wrote:
> >>>>>> 
> >>>>>>> I have that Marty. I see the http outbound request coming
in on the
> >>>>> guest
> >>>>>>> interface of the VR,and see the http request being sent
out on the
> >>>>> public
> >>>>>>> interface of the VR.
> >>>>>>> The traffic is flowing fine from guest to the outbound i/f
of the
> >>> VR.
> >>>>>>> This is tcpdump on the public i/f while guest is doing wget
to
> >>>>>>> 6x.xxx.xxx.xxx
> >>>>>>> 
> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00,
ethertype
> >>> IPv4
> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80:
Flags
> >>> [S],
> >>>>> seq
> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348
ecr
> >>>>>>> 0,nop,wscale 4], length 0  0x0000:  4500 003c ad1d 4000
3f06 2d13
> >>> 0a0b
> >>>>> 4fb2
> >>>>>>>       0x0010:  416e c660 98a2 0050 6ed2 de56 0000 0000
> >>> 0x0020:
> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a        0x0030: 
01a3 7444
> >>> 0000
> >>>>>>> 0000 0103 0304
> >>>>>>> 
> >>>>>>> 
> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100
> >>>>>>>> Subject: Re: Advanced Network - SNAT not working
> >>>>>>>> From: msweet.dev@gmail.com
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> 
> >>>>>>>> Hi Noel,
> >>>>>>>> 
> >>>>>>>> Can you run a tcpdump on both VR interfaces, this should
make it
> >>>>> apparent
> >>>>>>>> what is happening?
> >>>>>>>> 
> >>>>>>>> Thanks,
> >>>>>>>> Marty
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall <
> >>>>> noeldkendall@hotmail.com
> >>>>>>>> wrote:
> >>>>>>>> 
> >>>>>>>>> http://pastebin.com/3FZmFnvZ
> >>>>>>>>> Many thanks Marty.
> >>>>>>>>> Noel
> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100
> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working
> >>>>>>>>>> From: msweet.dev@gmail.com
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> 
> >>>>>>>>>> Hi Noel,
> >>>>>>>>>> 
> >>>>>>>>>> Could you put the IP tables on pastebin? GMail
has collapsed
> >>> the
> >>>>>>> lines
> >>>>>>>>>> horrifically.
> >>>>>>>>>> Have you also tried a tcpdump on both interfaces
on the VR?
> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be
called
> >>>>>>>>>> 
> >>>>>>>>>> I would expect worse connectivity if it was
a pure NAT issue,
> >>>>> but I
> >>>>>>> will
> >>>>>>>>>> review the tables later.
> >>>>>>>>>> 
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Marty
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall
<
> >>>>>>> noeldkendall@hotmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>> 
> >>>>>>>>>>> Not seeing return packets on VR. Suspect,
therefore, that
> >>> SNAT
> >>>>> is
> >>>>>>>>> fouled
> >>>>>>>>>>> up in some way.I have been doing wget to
from guest, can
> >>> see
> >>>>> the
> >>>>>>>>> outgoing
> >>>>>>>>>>> request fine, both in the guest andthe VR.
> >>>>>>>>>>> Could it be that the SNAT table entries
from the
> >>>>> 10.11.0.0/16subnet
> >>>>>>>>> to
> >>>>>>>>>>> dpt www are interfering withthe SNAT to
public ip?? (wild
> >>>>> guess) -
> >>>>>>> not
> >>>>>>>>> an
> >>>>>>>>>>> iptables expert by any stretch of the imagination
> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178
is the
> >>> guest
> >>>>> IP on
> >>>>>>>>> guest
> >>>>>>>>>>> network
> >>>>>>>>>>> iptables _L -t nat on the VR shows...
> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target 
   prot opt source
> >>>>>>>>>>> destination         DNAT       tcp  -- 
anywhere
> >>>>>>> anywhere
> >>>>>>>>>>>     tcp dpt:domain to:10.11.0.1 DNAT   
   tcp  --
> >>> anywhere
> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:www to:10.11.79.178:80
DNAT
> >>>>>>> tcp  --
> >>>>>>>>>>> anywhere             67.xxx.xxx.56     
  tcp dpt:www
> >>>>>>>>> to:10.11.79.178:80DNAT       tcp  --  anywhere
> >>>>>>> 67.xxx.xxx.56
> >>>>>>>>>       tcp dpt:https
> >>>>>>>>>>> to:10.11.79.178:443 DNAT       tcp  -- 
anywhere
> >>>>>>>>>>> 67.xxx.xxx.56        tcp dpt:https to:10.11.79.178:443DNAT
> >>>>>>> tcp
> >>>>>>>>> --
> >>>>>>>>>>> anywhere             67.xxx.xxx.56     
  tcp dpt:ssh
> >>>>>>>>> to:10.11.79.178:22DNAT       tcp  --  anywhere
> >>>>>>> 67.xxx.xxx.56
> >>>>>>>>>       tcp dpt:ssh
> >>>>>>>>>>> to:10.11.79.178:22 DNAT       tcp  --  anywhere
> >>>>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>       tcp dpt:ftp to:10.11.79.178:21 DNAT
      tcp  --
> >>>>> anywhere
> >>>>>>>>>>>      67.xxx.xxx.56        tcp dpt:ftp to:10.11.79.178:21DNAT
> >>>>>>>>> tcp
> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56 
      tcp
> >>> dpt:5901 to:
> >>>>>>>>>>> 10.11.79.178:5901 DNAT       tcp  --  anywhere
> >>>>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>       tcp dpt:5901 to:10.11.79.178:5901
> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target
    prot opt source
> >>>>>>>>>>> destination         SNAT       all  -- 
anywhere
> >>>>>>> anywhere
> >>>>>>>>>>>     to:67.xxx.xxx.56  SNAT       all  --
 anywhere
> >>>>>>>>> anywhere
> >>>>>>>>>>>         to:67.xxx.xxx.56  SNAT       all
 --  anywhere
> >>>>>>>>>>> anywhere            to:67.xxx.xxx.56 SNAT
      all  --
> >>>>> anywhere
> >>>>>>>>>>>  anywhere            to:67.xxx.xxx.56 SNAT
      all  --
> >>>>> anywhere
> >>>>>>>>>>>    anywhere            to:67.xxx.xxx.56SNAT
      all  --
> >>>>>>> anywhere
> >>>>>>>>>>>      anywhere            to:67.xxx.xxx.56
SNAT       all
> >>> --
> >>>>>>> anywhere
> >>>>>>>>>>>        anywhere            to:67.xxx.xxx.56
SNAT
> >>> all  --
> >>>>>>>>> anywhere
> >>>>>>>>>>>          anywhere            to:67.xxx.xxx.56
SNAT
> >>> tcp
> >>>>> --
> >>>>>>>>>>> 10.11.0.0/16         myguest           
 tcp dpt:www
> >>>>> to:10.11.0.1
> >>>>>>> SNAT
> >>>>>>>>>>>    tcp  --  10.11.0.0/16         myguest
            tcp
> >>>>>>> dpt:https
> >>>>>>>>>>> to:10.11.0.1 SNAT       tcp  --  10.11.0.0/16
> >>> myguest
> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT       tcp
 --  10.11.0.0/16
> >>>>>>>>> myguest
> >>>>>>>>>>>            tcp dpt:ftp to:10.11.0.1 SNAT
      tcp  --
> >>>>>>> 10.11.0.0/16
> >>>>>>>>>>>    myguest             tcp dpt:5901 to:10.11.0.1
SNAT
> >>>>> all
> >>>>>>> --
> >>>>>>>>>>> anywhere             anywhere          
 to:67.xxx.xxx.56
> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target     prot
opt source
> >>>>>>>>>>> destination         DNAT       tcp  -- 
anywhere
> >>>>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>      tcp dpt:www to:10.11.79.178:80 DNAT
      tcp  --
> >>>>> anywhere
> >>>>>>>>>>>    67.xxx.xxx.56       tcp dpt:https to:10.11.79.178:443DNAT
> >>>>>>>>> tcp
> >>>>>>>>>>> --  anywhere             67.xxx.xxx.56 
     tcp dpt:ssh
> >>> to:
> >>>>>>>>>>> 10.11.79.178:22 DNAT       tcp  --  anywhere
> >>>>>>> 67.xxx.xxx.56
> >>>>>>>>>>>    tcp dpt:ftp to:10.11.79.178:21 DNAT 
     tcp  --
> >>>>> anywhere
> >>>>>>>>>>>  67.xxx.xxx.56       tcp dpt:5901 to:10.11.79.178:5901
> >>>>>>>>>>> 
> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100
> >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT
not working
> >>>>>>>>>>>> From: msweet.dev@gmail.com
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Hi Noel,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Can you try using telnet to connect
to an external
> >>> webserver?
> >>>>>>> telnet
> >>>>>>>>>>>> www.google.com 80
> >>>>>>>>>>>> Can you also clarify: do you see the
response packets
> >>> reach
> >>>>> the
> >>>>>>> VR
> >>>>>>>>> and/or
> >>>>>>>>>>>> on what interfaces?
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Marty
> >>>>>>>>>>>> 
> >>>>>>>>>>>> On Saturday, September 14, 2013, Noel
Kendall wrote:
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> Guest OS cannot receive responses
to http GETs from
> >>>>> resources
> >>>>>>> on
> >>>>>>>>> the
> >>>>>>>>>>>>> Internet.
> >>>>>>>>>>>>> Network is advanced, VLAN isolated.
> >>>>>>>>>>>>> What is working:
> >>>>>>>>>>>>> - can browse guest website from
internet- can ssh to
> >>> guest
> >>>>> from
> >>>>>>>>>>> internet-
> >>>>>>>>>>>>> can VPN to guest network from internet
> >>>>>>>>>>>>> - network VR can access internet
sites no problem
> >>>>>>>>>>>>> What is not working:
> >>>>>>>>>>>>> - guest http traffic to external
website gets to VR on
> >>>>> internal
> >>>>>>>>> NIC,
> >>>>>>>>>>>>> packets forwarded to external site
via external NIC
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Response traffic is not seen. Appears
to be dropped.
> >>>>>>>>>>>>> Have been looking hard at IPTABLES
rules, doing
> >>> tcpdumps,
> >>>>> etc.
> >>>>>>>>>>>>> Am at this point stumped.
> >>>>>>>>>>>>> Any ideas on what could be wrong,
or how to determine
> >>> what
> >>>>>>> could be
> >>>>>>>>>>> wrong?
> >>>>>>>>>>>>> Thanks in advance everyone who tries
to help!
> >>>>>>>>>>>>> N.
> >>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>> 
> >>>>> 
> >>> 
> >>> 
> > 		 	   		  
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message