Return-Path: X-Original-To: apmail-cloudstack-users-archive@www.apache.org Delivered-To: apmail-cloudstack-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D3F02106CC for ; Mon, 16 Sep 2013 18:01:11 +0000 (UTC) Received: (qmail 61308 invoked by uid 500); 16 Sep 2013 18:01:11 -0000 Delivered-To: apmail-cloudstack-users-archive@cloudstack.apache.org Received: (qmail 61155 invoked by uid 500); 16 Sep 2013 18:01:09 -0000 Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@cloudstack.apache.org Delivered-To: mailing list users@cloudstack.apache.org Received: (qmail 61145 invoked by uid 99); 16 Sep 2013 18:01:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Sep 2013 18:01:07 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Chiradeep.Vittal@citrix.com designates 66.165.176.63 as permitted sender) Received: from [66.165.176.63] (HELO SMTP02.CITRIX.COM) (66.165.176.63) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Sep 2013 18:00:53 +0000 X-IronPort-AV: E=Sophos;i="4.90,917,1371081600"; d="scan'208";a="51964202" Received: from sjcpex01cl03.citrite.net ([10.216.14.145]) by FTLPIPO02.CITRIX.COM with ESMTP/TLS/AES128-SHA; 16 Sep 2013 18:00:31 +0000 Received: from SJCPEX01CL02.citrite.net ([169.254.2.131]) by SJCPEX01CL03.citrite.net ([10.216.14.145]) with mapi id 14.02.0342.004; Mon, 16 Sep 2013 11:00:30 -0700 From: Chiradeep Vittal To: "users@cloudstack.apache.org" Subject: Re: Advanced Network - SNAT not working Thread-Topic: Advanced Network - SNAT not working Thread-Index: AQHOsWOIEA5GsbPN3Ui6Yt+D/3rKCpnF4IwAgAAIa4CAAAOCgIAACXCAgAANd4CAAA3wAIABG78AgAAOFICAAEYCgIAAIHCAgAATFYCAAAsgAIAAiiGAgAB4rQD//+figA== Date: Mon, 16 Sep 2013 18:00:29 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.6.130613 x-originating-ip: [10.216.48.12] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Suggest that you stop and start (not reboot) the router from the Admin GUI. On 9/16/13 5:26 AM, "Noel Kendall" wrote: >Jayapal, I did a ping test and traced as you suggested. tcpdump >monitoring was done on the public facing interface of the VR. >From within the VR, ping to public IP functions correctly, source address >is the public IP assigned to the VR. >From within the guest, ping to same public IP, does not function, source >address is (as you suspected) the IP of guest on the guest network of VR. >Therefore, it must be: the SNAT rule in iptables in the VR is being >bypassed... that is, the packets are being forwarded without SNAT being >performed on them correctly. >Noel > >> From: jayapalreddy.uradi@citrix.com >> To: users@cloudstack.apache.org >> Subject: Re: Advanced Network - SNAT not working >> Date: Mon, 16 Sep 2013 05:14:53 +0000 >>=20 >> Hi, >>=20 >> I think when the packets are going out the packets are NATed with >>private ip, that can't reach back to router. >> From the VR when you ping public network observe with what source ip >>address the packet is going out and >> From the guest VM when you access public n/w observe on VR with what >>source ip the packet is going out. >> In later case I think the source ip address is different. >>=20 >> Thanks, >> Jayapal >>=20 >>=20 >> On 16-Sep-2013, at 2:30 AM, Noel Kendall >>wrote: >>=20 >> > No other NAT. There is nothing but copper between the KVM host >>machine and the ISP router.There is an L2/L3 switch that the packets >>travel through. However, there is no forwarding in the switch,just >>straight through. I've had a well-functioning V4.0.1 environment running >>on this same configurationin the past. What is new is the conversion to >>4.1 (which was a clean install). >> > It's very mysterious, I have never seen anything like this before. >>There are two other VRs, both having same issue. >> > I will try your suggestion. >> > Noel >> >> Date: Sun, 15 Sep 2013 21:20:41 +0100 >> >> Subject: Re: Advanced Network - SNAT not working >> >> From: msweet.dev@gmail.com >> >> To: users@cloudstack.apache.org >> >>=20 >> >> This is mostly confusing that the packets are not seen on the VR >>public >> >> interface, seeing as other services are working. >> >> If it was a local NAT issue then the packet would atleast get into >>that >> >> interface. Do you have any upstream devices providing NAT? Or any >>other VR >> >> with the issue? >> >>=20 >> >> It may be worth recreating the VR, by stopping and destroying it and >> >> creating another guest to start a fresh. >> >>=20 >> >> Marty >> >>=20 >> >>=20 >> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall >>wrote: >> >>=20 >> >>> Marty, if I run a telnet 80 from a shell in the guest, >> >>> while running a tcpdumpon the public i/f of the VR: >> >>> - I can see the outbound packets going out- I do not see a response >>packet >> >>> coming back in >> >>> FYI there are no firewalls outbound from the KVM host. The host >>bridges vi >> >>> CS networkingdirectly out on to the internet via a switch. >> >>> Note that traffic from outside (ssh, web) can happily traverse the >>VR to >> >>> the guest. I get the usualits working html page from the guest. >>This tells >> >>> me that there is nothing outbound from the VR thatis filtering >>packets. >> >>> Am truly stumped. This is mysterious indeed. >> >>> From within the VR, can happily telnet to 80 and >>receive >> >>> response.Only if packet came from guest and was forwarded does the >>response >> >>> not show up. >> >>> In short: >> >>> wget from VR to www.xyz.com works, response received and saved >> >>> wget from guest to www.xyz.com does not work, network not available >> >>> displayed on guest, response packets not seen on the public i/f of >>VR at all >> >>> Noel >> >>>=20 >> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100 >> >>>> Subject: Re: Advanced Network - SNAT not working >> >>>> From: msweet.dev@gmail.com >> >>>> To: users@cloudstack.apache.org >> >>>>=20 >> >>>> Hi Noel, >> >>>>=20 >> >>>> Can you answer: Does the traffic come back on the public >>interface? and >> >>>> then onto the Guest interface? >> >>>>=20 >> >>>> Thanks, >> >>>> Marty >> >>>>=20 >> >>>>=20 >> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall >>> >>>> wrote: >> >>>>=20 >> >>>>> Indeed, yes, a wget executed on the VR to a public website works >>just >> >>> fine. >> >>>>> Noel >> >>>>>=20 >> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100 >> >>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>> From: msweet.dev@gmail.com >> >>>>>> To: users@cloudstack.apache.org >> >>>>>>=20 >> >>>>>> Hi Noel, >> >>>>>>=20 >> >>>>>> Does the traffic come back on the public interface? and then >>onto the >> >>>>> Guest >> >>>>>> interface? >> >>>>>>=20 >> >>>>>> Does a wget on the VR work? >> >>>>>>=20 >> >>>>>> Marty >> >>>>>>=20 >> >>>>>>=20 >> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall < >> >>> noeldkendall@hotmail.com >> >>>>>> wrote: >> >>>>>>=20 >> >>>>>>> I have that Marty. I see the http outbound request coming in on >>the >> >>>>> guest >> >>>>>>> interface of the VR,and see the http request being sent out on >>the >> >>>>> public >> >>>>>>> interface of the VR. >> >>>>>>> The traffic is flowing fine from guest to the outbound i/f of >>the >> >>> VR. >> >>>>>>> This is tcpdump on the public i/f while guest is doing wget to >> >>>>>>> 6x.xxx.xxx.xxx >> >>>>>>>=20 >> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00, ethertype >> >>> IPv4 >> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80: >>Flags >> >>> [S], >> >>>>> seq >> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348 >>ecr >> >>>>>>> 0,nop,wscale 4], length 0 0x0000: 4500 003c ad1d 4000 3f06 >>2d13 >> >>> 0a0b >> >>>>> 4fb2 >> >>>>>>> 0x0010: 416e c660 98a2 0050 6ed2 de56 0000 0000 >> >>> 0x0020: >> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a 0x0030: 01a3 >>7444 >> >>> 0000 >> >>>>>>> 0000 0103 0304 >> >>>>>>>=20 >> >>>>>>>=20 >> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100 >> >>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>>>> From: msweet.dev@gmail.com >> >>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>=20 >> >>>>>>>> Hi Noel, >> >>>>>>>>=20 >> >>>>>>>> Can you run a tcpdump on both VR interfaces, this should make >>it >> >>>>> apparent >> >>>>>>>> what is happening? >> >>>>>>>>=20 >> >>>>>>>> Thanks, >> >>>>>>>> Marty >> >>>>>>>>=20 >> >>>>>>>>=20 >> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall < >> >>>>> noeldkendall@hotmail.com >> >>>>>>>> wrote: >> >>>>>>>>=20 >> >>>>>>>>> http://pastebin.com/3FZmFnvZ >> >>>>>>>>> Many thanks Marty. >> >>>>>>>>> Noel >> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100 >> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>>>>>> From: msweet.dev@gmail.com >> >>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>>=20 >> >>>>>>>>>> Hi Noel, >> >>>>>>>>>>=20 >> >>>>>>>>>> Could you put the IP tables on pastebin? GMail has collapsed >> >>> the >> >>>>>>> lines >> >>>>>>>>>> horrifically. >> >>>>>>>>>> Have you also tried a tcpdump on both interfaces on the VR? >> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be called >> >>>>>>>>>>=20 >> >>>>>>>>>> I would expect worse connectivity if it was a pure NAT issue, >> >>>>> but I >> >>>>>>> will >> >>>>>>>>>> review the tables later. >> >>>>>>>>>>=20 >> >>>>>>>>>> Thanks, >> >>>>>>>>>> Marty >> >>>>>>>>>>=20 >> >>>>>>>>>>=20 >> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall < >> >>>>>>> noeldkendall@hotmail.com >> >>>>>>>>>> wrote: >> >>>>>>>>>>=20 >> >>>>>>>>>>> Not seeing return packets on VR. Suspect, therefore, that >> >>> SNAT >> >>>>> is >> >>>>>>>>> fouled >> >>>>>>>>>>> up in some way.I have been doing wget to from guest, can >> >>> see >> >>>>> the >> >>>>>>>>> outgoing >> >>>>>>>>>>> request fine, both in the guest andthe VR. >> >>>>>>>>>>> Could it be that the SNAT table entries from the >> >>>>> 10.11.0.0/16subnet >> >>>>>>>>> to >> >>>>>>>>>>> dpt www are interfering withthe SNAT to public ip?? (wild >> >>>>> guess) - >> >>>>>>> not >> >>>>>>>>> an >> >>>>>>>>>>> iptables expert by any stretch of the imagination >> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178 is the >> >>> guest >> >>>>> IP on >> >>>>>>>>> guest >> >>>>>>>>>>> network >> >>>>>>>>>>> iptables _L -t nat on the VR shows... >> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target prot opt source >> >>>>>>>>>>> destination DNAT tcp -- anywhere >> >>>>>>> anywhere >> >>>>>>>>>>> tcp dpt:domain to:10.11.0.1 DNAT tcp -- >> >>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:www to:10.11.79.178:80 DNAT >> >>>>>>> tcp -- >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:www >> >>>>>>>>> to:10.11.79.178:80DNAT tcp -- anywhere >> >>>>>>> 67.xxx.xxx.56 >> >>>>>>>>> tcp dpt:https >> >>>>>>>>>>> to:10.11.79.178:443 DNAT tcp -- anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT >> >>>>>>> tcp >> >>>>>>>>> -- >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:ssh >> >>>>>>>>> to:10.11.79.178:22DNAT tcp -- anywhere >> >>>>>>> 67.xxx.xxx.56 >> >>>>>>>>> tcp dpt:ssh >> >>>>>>>>>>> to:10.11.79.178:22 DNAT tcp -- anywhere >> >>>>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- >> >>>>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:ftp to:10.11.79.178:21DNAT >> >>>>>>>>> tcp >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp >> >>> dpt:5901 to: >> >>>>>>>>>>> 10.11.79.178:5901 DNAT tcp -- anywhere >> >>>>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:5901 to:10.11.79.178:5901 >> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target prot opt source >> >>>>>>>>>>> destination SNAT all -- anywhere >> >>>>>>> anywhere >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere >> >>>>>>>>> anywhere >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- >> >>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- >> >>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56SNAT all -- >> >>>>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all >> >>> -- >> >>>>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT >> >>> all -- >> >>>>>>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT >> >>> tcp >> >>>>> -- >> >>>>>>>>>>> 10.11.0.0/16 myguest tcp dpt:www >> >>>>> to:10.11.0.1 >> >>>>>>> SNAT >> >>>>>>>>>>> tcp -- 10.11.0.0/16 myguest tcp >> >>>>>>> dpt:https >> >>>>>>>>>>> to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 >> >>> myguest >> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 >> >>>>>>>>> myguest >> >>>>>>>>>>> tcp dpt:ftp to:10.11.0.1 SNAT tcp -- >> >>>>>>> 10.11.0.0/16 >> >>>>>>>>>>> myguest tcp dpt:5901 to:10.11.0.1 SNAT >> >>>>> all >> >>>>>>> -- >> >>>>>>>>>>> anywhere anywhere to:67.xxx.xxx.56 >> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target prot opt source >> >>>>>>>>>>> destination DNAT tcp -- anywhere >> >>>>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:www to:10.11.79.178:80 DNAT tcp -- >> >>>>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT >> >>>>>>>>> tcp >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp dpt:ssh >> >>> to: >> >>>>>>>>>>> 10.11.79.178:22 DNAT tcp -- anywhere >> >>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- >> >>>>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:5901 to:10.11.79.178:5901 >> >>>>>>>>>>>=20 >> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100 >> >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>>>>>>>> From: msweet.dev@gmail.com >> >>>>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>>>>=20 >> >>>>>>>>>>>> Hi Noel, >> >>>>>>>>>>>>=20 >> >>>>>>>>>>>> Can you try using telnet to connect to an external >> >>> webserver? >> >>>>>>> telnet >> >>>>>>>>>>>> www.google.com 80 >> >>>>>>>>>>>> Can you also clarify: do you see the response packets >> >>> reach >> >>>>> the >> >>>>>>> VR >> >>>>>>>>> and/or >> >>>>>>>>>>>> on what interfaces? >> >>>>>>>>>>>>=20 >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> Marty >> >>>>>>>>>>>>=20 >> >>>>>>>>>>>> On Saturday, September 14, 2013, Noel Kendall wrote: >> >>>>>>>>>>>>=20 >> >>>>>>>>>>>>> Guest OS cannot receive responses to http GETs from >> >>>>> resources >> >>>>>>> on >> >>>>>>>>> the >> >>>>>>>>>>>>> Internet. >> >>>>>>>>>>>>> Network is advanced, VLAN isolated. >> >>>>>>>>>>>>> What is working: >> >>>>>>>>>>>>> - can browse guest website from internet- can ssh to >> >>> guest >> >>>>> from >> >>>>>>>>>>> internet- >> >>>>>>>>>>>>> can VPN to guest network from internet >> >>>>>>>>>>>>> - network VR can access internet sites no problem >> >>>>>>>>>>>>> What is not working: >> >>>>>>>>>>>>> - guest http traffic to external website gets to VR on >> >>>>> internal >> >>>>>>>>> NIC, >> >>>>>>>>>>>>> packets forwarded to external site via external NIC >> >>>>>>>>>>>>>=20 >> >>>>>>>>>>>>> Response traffic is not seen. Appears to be dropped. >> >>>>>>>>>>>>> Have been looking hard at IPTABLES rules, doing >> >>> tcpdumps, >> >>>>> etc. >> >>>>>>>>>>>>> Am at this point stumped. >> >>>>>>>>>>>>> Any ideas on what could be wrong, or how to determine >> >>> what >> >>>>>>> could be >> >>>>>>>>>>> wrong? >> >>>>>>>>>>>>> Thanks in advance everyone who tries to help! >> >>>>>>>>>>>>> N. >> >>>>>>>>>>>>>=20 >> >>>>>>>>>>>=20 >> >>>>>>>>>>>=20 >> >>>>>>>>>=20 >> >>>>>>>>>=20 >> >>>>>>>=20 >> >>>>>>>=20 >> >>>>>=20 >> >>>>>=20 >> >>>=20 >> >>>=20 >> > =20 >>=20 > =20