Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D2A4210F1B for ; Wed, 28 May 2014 19:24:21 +0000 (UTC) Received: (qmail 18902 invoked by uid 500); 28 May 2014 19:24:21 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 18857 invoked by uid 500); 28 May 2014 19:24:21 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 18849 invoked by uid 99); 28 May 2014 19:24:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 May 2014 19:24:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of andrija.panic@gmail.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 May 2014 19:24:17 +0000 Received: by mail-ie0-f179.google.com with SMTP id rd18so10893130iec.10 for ; Wed, 28 May 2014 12:23:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=tan2zTxfVqgEkUxSK+KfUfdJHG/+hBCsMw5mBRsw7UA=; b=bwt9srVS3VCIesty1UHPGcyRacvgo3UiOwIlqouz2lt3/Q2YnEL3ommKOq83UpWq72 3shBVj+17RdL8LjJIkjYvVyDicZEhL1DOWg8egIGDF3nT4aoqbkLvsaXhEAhhnmpSAch YCUjgoxKrcIdPc3Se2AV62GLxbD0xFZgKD7nH+ylegCJSD3QQEhA07M/2w0HOPeFYaY6 4FLi0drCVQziZ7W33TIBZwK0StP/p1fWlWWitogKKmxG6PZZVs14IgZAfS+4HTnMp19J o98GACEZDGoBrU6UnYNH/qUKGAfm8bG2pCml7Fq/6HsJkl7Edx7EdEn2Hidw01BVl6Rh Bbhg== MIME-Version: 1.0 X-Received: by 10.50.20.8 with SMTP id j8mr3998582ige.17.1401305033380; Wed, 28 May 2014 12:23:53 -0700 (PDT) Received: by 10.42.27.74 with HTTP; Wed, 28 May 2014 12:23:53 -0700 (PDT) In-Reply-To: References: Date: Wed, 28 May 2014 21:23:53 +0200 Message-ID: Subject: Re: VPC's VR missing public NIC eth1 From: Andrija Panic To: "dev@cloudstack.apache.org" Content-Type: multipart/alternative; boundary=047d7bd75266bcbecd04fa7ac068 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bd75266bcbecd04fa7ac068 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Daan, I don't think this is my issue, at least I don't make use of private gateway - this is just simple as: create new VPC from scratch - Public IP is not assigned to VR eth1 interface inside VR... I have filed the bug: https://issues.apache.org/jira/browse/CLOUDSTACK-6801 This same thing happened previously to Andrei Mikhailovsky: http://mail-archives.apache.org/mod_mbox/cloudstack-users/201405.mbox/%3C33= 347835.250.1399336340785.JavaMail.andrei@tuchka%3Eand it is not resolved Thanks, Andrija On 28 May 2014 21:01, Daan Hoogland wrote: > Andrija, > > this sound like something we seen as well. > can you check if this is it : > https://issues.apache.org/jira/browse/CLOUDSTACK-6485 > > thanks, > Daan > > On Wed, May 28, 2014 at 3:30 PM, Andrija Panic > wrote: > > Hi there, > > > > I'm having big time problems with Public IP missing from VPC VR's eth1, > > after upgrade to ACS 4.3.1 - did not found this filed as bug so far...a= nd > > it worked all fine on ACS 4.2.1. > > > > No help so far from user mailing list... > > > > Below is a detailed explanation, and logs from inside VR, and from > > management (all fine with management logs...) > > > > If anybody can help, I would very much appriciate this, since now I ha= ve > > bunch fo VPC unoperational... > > > > Thanks > > > > ---------- Forwarded message ---------- > > From: Andrija Panic > > Date: 28 May 2014 14:50 > > Subject: Re: VPC's VR missing public NIC eth1 > > To: users@cloudstack.apache.org > > > > > > and as I said eth1 is present: > > > > root@r-794-VM:~# cat /proc/net/dev > > Inter-| Receive | > Transmit > > face |bytes packets errs drop fifo frame compressed multicast|bytes > > packets errs drop fifo colls carrier compressed > > eth3: 11484 131 0 0 0 0 0 0 > 11590 > > 131 0 0 0 0 0 0 > > lo: 214 2 0 0 0 0 0 0 > 214 > > 2 0 0 0 0 0 0 > > eth2: 32970 544 0 0 0 0 0 0 > 2084 > > 24 0 0 0 0 0 0 > > eth1: 0 0 0 0 0 0 0 0 > 0 > > 0 0 0 0 0 0 0 > > eth0: 150207 1319 0 0 0 0 0 0 > 264232 > > 1180 0 0 0 0 0 0 > > > > > > On 28 May 2014 14:47, Andrija Panic wrote: > > > >> Also, from /var/log/messages/ inside VR: > >> > >> This is a major show stopper - all our VPCs are unusable complete. > >> Anybody... ? > >> > >> May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 0 seconds > >> May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 1 seconds > >> May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 2 seconds > >> May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 3 seconds > >> May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 4 seconds > >> May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 5 seconds > >> May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 6 seconds > >> May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 7 seconds > >> May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 8 seconds > >> May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 9 seconds > >> May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 10 seconds > >> May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 11 seconds > >> May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 12 seconds > >> May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 13 seconds > >> May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 14 seconds > >> May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 15 seconds > >> May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 16 seconds > >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull never > >> appeared > >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip 46.232.180.24= 6 > on > >> interface ethnull > >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing > 46.232.180.246 > >> on interface ethnull > >> May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT > >> 46.232.180.246 on interface ethnull > >> May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT > 46.232.180.246 > >> on interface eth1 > >> > >> > >> > >> > >> On 28 May 2014 12:59, Andrija Panic wrote: > >> > >>> Defined eth1 manually inside /etc/network/interfaces inside VPC's VR. > >>> iface eth1 inet static > >>> address 46.232.180.246 > >>> netmask 255.255.255.0 > >>> > >>> ifup eth1 > >>> ip route add default via 46.232.180.1 > >>> > >>> so now VR works fine (have access to internet) > >>> > >>> But again, adding new IP to VR, and enabling static NAT is failing... > >>> That is, geting new IP works fine (just associated with account) > >>> But enabling static NAT fails, due to "resource unavailable" > >>> > >>> Here are management logs: > >>> 2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl] > >>> (catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static > nat > >>> rule due to > >>> com.cloud.exception.ResourceUnavailableException: Resource > [DataCenter:1] > >>> is unreachable: Unable to apply static nat rules on router > >>> at > >>> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(Vi= rtualNetworkApplianceManagerImpl.java:3915) > >>> at > >>> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNa= ts(VirtualNetworkApplianceManagerImpl.java:3963) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method= ) > >>> at > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :57) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:622) > >>> at > >>> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(A= opUtils.java:317) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpo= int(ReflectiveMethodInvocation.java:183) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl= ectiveMethodInvocation.java:150) > >>> at > >>> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex= poseInvocationInterceptor.java:91) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl= ectiveMethodInvocation.java:172) > >>> at > >>> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAop= Proxy.java:204) > >>> at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source) > >>> at > >>> > com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRou= terElement.java:650) > >>> at > >>> > com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerIm= pl.java:1762) > >>> at > >>> > com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManager= Impl.java:1324) > >>> at > >>> > com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl= .java:602) > >>> at > >>> > com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl= .java:446) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method= ) > >>> at > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :57) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:622) > >>> at > >>> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(A= opUtils.java:317) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpo= int(ReflectiveMethodInvocation.java:183) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl= ectiveMethodInvocation.java:150) > >>> at > >>> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java= :50) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl= ectiveMethodInvocation.java:161) > >>> at > >>> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex= poseInvocationInterceptor.java:91) > >>> at > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl= ectiveMethodInvocation.java:172) > >>> at > >>> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAop= Proxy.java:204) > >>> at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source) > >>> at > >>> > org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(Ena= bleStaticNatCmd.java:129) > >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:16= 1) > >>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531) > >>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374) > >>> at > >>> com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323) > >>> at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53) > >>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115) > >>> at > >>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(D= efaultManagedContext.java:56) > >>> at > >>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWith= Context(DefaultManagedContext.java:103) > >>> at > >>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithC= ontext(DefaultManagedContext.java:53) > >>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:11= 2) > >>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74) > >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:61= 7) > >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:71= 7) > >>> at > >>> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat= ionFilterChain.java:290) > >>> at > >>> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte= rChain.java:206) > >>> at > >>> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve= .java:233) > >>> at > >>> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve= .java:191) > >>> at > >>> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:= 127) > >>> at > >>> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:= 102) > >>> at > >>> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > >>> at > >>> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j= ava:109) > >>> at > >>> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29= 8) > >>> at > >>> > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.ja= va:889) > >>> at > >>> > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.proces= s(Http11NioProtocol.java:721) > >>> at > >>> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.ja= va:2274) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1146) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:615) > >>> at java.lang.Thread.run(Thread.java:701) > >>> > >>> > >>> > >>> > >>> On 28 May 2014 00:58, Andrija Panic wrote: > >>> > >>>> Hi Jayapal, > >>>> > >>>> eth1 seems present (lspci and virsh comfirmed), but not started insi= de > >>>> VPC's VR - (VR used for Shared Network is fine)... > >>>> I could confirm by virsh that is is plugged inside appropriate bridg= e > >>>> breth1-500 (check management logs...) > >>>> > >>>> management log while createing new VPC (VR) - > >>>> http://pastebin.com/s77nu5Ei > >>>> The public IP is there, so command is fine for creating it I guess..= . > >>>> > >>>> VR's /var/log/cloud.log after rebooting VR from CS GUI > >>>> Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config > >>>> Tue May 27 22:46:58 UTC 2014 Detected that we are running inside kvm > >>>> guest > >>>> Tue May 27 22:46:59 UTC 2014 Patching cloud service > >>>> Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml > >>>> Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system vm > >>>> Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP > >>>> Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC > >>>> Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq =3D 1 > >>>> Tue May 27 22:47:00 UTC 2014 Enable service haproxy =3D 1 > >>>> Tue May 27 22:47:00 UTC 2014 Processors =3D 1 Enable service =3D 0 > >>>> Tue May 27 22:47:00 UTC 2014 Enable service cloud =3D 0 > >>>> Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter > >>>> Tue May 27 22:47:00 UTC 2014 disable rpfilter > >>>> Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding =3D 1 > >>>> Tue May 27 22:47:00 UTC 2014 enable_fwding =3D 1 > >>>> > >>>> ifconfig (no eth1 shown) > >>>> > >>>> eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c > >>>> inet addr:169.254.3.92 Bcast:169.254.255.255 > Mask:255.255.0.0 > >>>> > >>>> eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10 > >>>> inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0 > >>>> > >>>> eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05 > >>>> inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0 > >>>> > >>>> lo Link encap:Local Loopback > >>>> inet addr:127.0.0.1 Mask:255.0.0.0 > >>>> > >>>> > >>>> cat /etc/network/interfaces > >>>> auto lo eth0 > >>>> iface lo inet loopback > >>>> iface eth0 inet static > >>>> address 169.254.3.92 > >>>> netmask 255.255.0.0 > >>>> > >>>> lspci - shows 4 ehternet addapters > >>>> ethtool eth1 =3D no link detected > >>>> virsh - confirmed that eth1 is plugged to correct bridge (breth1-500= ) > as > >>>> indicated by management logs, and shows good MAC address as shown in > >>>> managemetn log on pastebin.. > >>>> > >>>> This is completely makeing VPCs unusable... > >>>> :( > >>>> > >>>> Cheers > >>>> > >>>> > >>>> On 27 May 2014 16:36, Jayapal Reddy Uradi < > jayapalreddy.uradi@citrix.com > >>>> > wrote: > >>>> > >>>>> Hi, > >>>>> Can you please share management server and router logs in > pastebin.comto understand the issue ? > >>>>> > >>>>> Thanks, > >>>>> Jayapal > >>>>> > >>>>> On 27-May-2014, at 6:21 PM, Andrija Panic > >>>>> wrote: > >>>>> > >>>>> > Hi, > >>>>> > > >>>>> > after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC lo= st > >>>>> their > >>>>> > eth1 which is public NIC. VR got eth0(control nic) and eth2 and > eth3 > >>>>> (bith > >>>>> > belonging to Tiers). From CS GUI, it is reported that the VR has > eth1 > >>>>> with > >>>>> > Public network attached, but from inside (ssh to VR) there is no > eth1 > >>>>> with > >>>>> > public IP... > >>>>> > > >>>>> > Even after destroying those VR, they are recreated again, but > without > >>>>> eth1. > >>>>> > > >>>>> > Anybody experienced same situtation ? > >>>>> > > >>>>> > Thanks, > >>>>> > > >>>>> > -- > >>>>> > > >>>>> > Andrija Pani=C4=87 > >>>>> > -------------------------------------- > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> > >>>> > > > > -- > Daan > --=20 Andrija Pani=C4=87 -------------------------------------- http://admintweets.com -------------------------------------- --047d7bd75266bcbecd04fa7ac068--