cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cs user <acldstk...@gmail.com>
Subject Re: Virtual Routers not responding to dns requests
Date Fri, 27 Mar 2015 08:57:54 GMT
Hi Jayapal,

Those two parameters are both set to 1.

We have a router which has survived the upgrade, and is still able to
receive and return dns requests from instances. We have checked on the
xenserver host and can see the following iptable config:

Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination
> BRIDGE-FIREWALL  all  --  anywhere             anywhere            PHYSDEV
> match --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth0+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond3+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond0+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth1+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth3+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth6+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond1+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth2+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth5+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth7+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth4+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond2+ --physdev-is-bridged
> DROP       all  --  anywhere             anywhere



However, on a xenserver which is running a router which is not working, we
can see the following:

Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination
> BRIDGE-FIREWALL  all  --  anywhere             anywhere            PHYSDEV
> match --physdev-is-bridged
>
ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth1+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth4+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth3+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth7+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond3+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond0+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth6+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond1+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth5+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond2+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth0+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth2+ --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth1 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth4 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth3 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth7 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond3 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth6 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond1 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth5 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond2 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth0 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out eth2 --physdev-is-bridged
> ACCEPT     all  --  anywhere             anywhere            PHYSDEV match
> --physdev-out bond0 --physdev-is-bridged
> DROP       all  --  anywhere             anywhere


It seems the config is duplicated but without the pluses. I think these are
similar to wildcard entries?

Cheers!







On Fri, Mar 27, 2015 at 8:46 AM, Jayapal Reddy Uradi <
jayapalreddy.uradi@citrix.com> wrote:

> Silly question but Is your xenserver configured bridge mode related
> settings correctly ?
>
> #xe-switch-network-backend bridge
> #echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
> #echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables
>
> Thanks,
> Jayapal
> On 27-Mar-2015, at 1:50 PM, cs user <acldstkusr@gmail.com<mailto:
> acldstkusr@gmail.com>> wrote:
>
> Hi Somesh,
>
> arping looks good, the correct mac address is displayed and we get a
> unicast reply from the ip address.
>
> Erik, tried restarting dnsmasq, all looks fine. VR is able to perform
> outgoing dns requests. There is nothing in syslog/dnsmasq logs that I can
> see. No egress rules are in place. The system vm's are able to perform
> dig's against google's dns, but not the virtual router. It seems it is
> being blocked at the xen level.
>
> We're seeing the below in the logs when restarting a network (either
> ticking clear config or not). This appears to be similar to :
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-7605
>
> We are using basic zones, some have multiple pods, others don't. We see the
> same error in both. The routers come up though and go green, and dnsmasq is
> populated with the relevant info. DNS lookups work locally on the router,
> just not remotely. DHCP is working for new machines which get spun up.
>
> Is there a way to debug this? I've checked the logs on the router
> (cloud.log) can't see any errors in there.
>
> 2015-03-27 08:12:45,081 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Implementing the
> network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] elements and
> resources as a part of network restart
>
> 2015-03-27 08:12:45,096 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking
> SecurityGroupProvider to implemenet
> Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
>
> 2015-03-27 08:12:45,103 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking
> VirtualRouter to implemenet
> Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
>
> 2015-03-27 08:12:45,112 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is acquired
> for network id 204 as a part of router startup in
>
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> : Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()]
>
> 2015-03-27 08:12:45,119 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Skipping VR
> deployment: Found a running or starting VR in Pod null id=8
>
> 2015-03-27 08:12:45,120 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is released
> for network id 204 as a part of router startup in
>
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> : Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()]
>
> 2015-03-27 08:12:45,123 WARN  [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Failed to
> implement network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
> elements and resources as a part of network restart due to
>
> java.lang.NullPointerException
>
>        at
>
> com.cloud.network.element.VirtualRouterElement.getRouters(VirtualRouterElement.java:952)
>
>        at
>
> com.cloud.network.element.VirtualRouterElement.prepareAggregatedExecution(VirtualRouterElement.java:1099)
>
>        at
>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1090)
>
>        at
>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
>
>        at
>
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
>
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>        at java.lang.reflect.Method.invoke(Method.java:601)
>
>        at
>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>
>        at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>
>        at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>
>        at
>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>
>        at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>
>        at
>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>
>        at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>
>        at
>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>
>        at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>
>        at
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>
>        at $Proxy156.restartNetwork(Unknown Source)
>
>        at
>
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
>
>        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>
>        at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>
>        at
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>
>        at
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
>        at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
>        at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
>        at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
>        at
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
>        at
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>
>        at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
>        at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
>        at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
>        at java.lang.Thread.run(Thread.java:722)
>
> 2015-03-27 08:12:45,125 WARN  [c.c.n.NetworkServiceImpl]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Network id=204
> failed to restart.
>
> 2015-03-27 08:12:45,140 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235) Complete async job-189235,
> jobStatus: FAILED, resultCode: 530,
>
> result:org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to restart network"}
>
> 2015-03-27 08:12:45,152 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235) Done executing
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
> job-189235
>
> 2015-03-27 08:12:45,158 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-16:ctx-0b0aa78a job-189235) Remove job-189235 from job
> monitoring
>
>
>
>
>
> On Thu, Mar 26, 2015 at 7:38 PM, Somesh Naidu <Somesh.Naidu@citrix.com>
> wrote:
>
> You might want to do a arpping on the router's IP from one of the guests
> and see how many records are returned.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Erik Weber [mailto:terbolous@gmail.com]
> Sent: Thursday, March 26, 2015 12:54 PM
> To: users@cloudstack.apache.org
> Subject: Re: Virtual Routers not responding to dns requests
>
> I briefly remember having similar problems at some point, but do not recall
> details as version nor the solution.
>
> 1) Does it work if you restart dnsmasq on the VR?
> 2) is the VR able to do outgoing dns requests?
> 3) anything in syslog/dnsmasq logs?
> 4) any egress rules in place?
>
>
> Erik
>
> Den torsdag 26. mars 2015 skrev cs user <acldstkusr@gmail.com> følgende:
>
> Hi All,
>
> We have upgraded from 4.3 to 4.4.2.
>
> After some issues with starting the systemvm's, the virtual routers no
> longer responding to dns requests from the vm's which we start (or
> existing
> ones).
>
> We have disabled the firewall on the virtual routers and ran tcpdump on
> them but we can't see any inbound traffic on port 53 (udp or tcp). If we
> log onto the virtual routers and dig locally against eth0 and the alias
> on
> eth0 both of these return fine with the correct IP.
>
> This is using Xenserver 6.1 as the host.
>
> Has anyone come across this before? dhcp lookups appear to be working
> fine.
> Is there a firewall rule in place on the router vm's (other than
> iptables),
> similar to the security groups which are applied by Xen, which is
> preventing these requests from hitting the routers?
>
> Many thanks for any help.
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message