cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayapal Reddy Uradi <jayapalreddy.ur...@citrix.com>
Subject Re: Virtual Routers not responding to dns requests
Date Fri, 27 Mar 2015 08:46:31 GMT
Silly question but Is your xenserver configured bridge mode related settings correctly ?

#xe-switch-network-backend bridge
#echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
#echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables

Thanks,
Jayapal
On 27-Mar-2015, at 1:50 PM, cs user <acldstkusr@gmail.com<mailto:acldstkusr@gmail.com>>
wrote:

Hi Somesh,

arping looks good, the correct mac address is displayed and we get a
unicast reply from the ip address.

Erik, tried restarting dnsmasq, all looks fine. VR is able to perform
outgoing dns requests. There is nothing in syslog/dnsmasq logs that I can
see. No egress rules are in place. The system vm's are able to perform
dig's against google's dns, but not the virtual router. It seems it is
being blocked at the xen level.

We're seeing the below in the logs when restarting a network (either
ticking clear config or not). This appears to be similar to :

https://issues.apache.org/jira/browse/CLOUDSTACK-7605

We are using basic zones, some have multiple pods, others don't. We see the
same error in both. The routers come up though and go green, and dnsmasq is
populated with the relevant info. DNS lookups work locally on the router,
just not remotely. DHCP is working for new machines which get spun up.

Is there a way to debug this? I've checked the logs on the router
(cloud.log) can't see any errors in there.

2015-03-27 08:12:45,081 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Implementing the
network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] elements and
resources as a part of network restart

2015-03-27 08:12:45,096 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking
SecurityGroupProvider to implemenet
Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]

2015-03-27 08:12:45,103 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking
VirtualRouter to implemenet
Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]

2015-03-27 08:12:45,112 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is acquired
for network id 204 as a part of router startup in
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()]

2015-03-27 08:12:45,119 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Skipping VR
deployment: Found a running or starting VR in Pod null id=8

2015-03-27 08:12:45,120 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is released
for network id 204 as a part of router startup in
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()]

2015-03-27 08:12:45,123 WARN  [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Failed to
implement network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
elements and resources as a part of network restart due to

java.lang.NullPointerException

       at
com.cloud.network.element.VirtualRouterElement.getRouters(VirtualRouterElement.java:952)

       at
com.cloud.network.element.VirtualRouterElement.prepareAggregatedExecution(VirtualRouterElement.java:1099)

       at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1090)

       at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)

       at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)

       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

       at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

       at java.lang.reflect.Method.invoke(Method.java:601)

       at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)

       at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

       at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

       at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)

       at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

       at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)

       at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

       at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

       at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

       at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

       at $Proxy156.restartNetwork(Unknown Source)

       at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)

       at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)

       at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)

       at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)

       at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)

       at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)

       at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)

       at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)

       at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)

       at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)

       at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

       at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

       at java.util.concurrent.FutureTask.run(FutureTask.java:166)

       at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

       at java.lang.Thread.run(Thread.java:722)

2015-03-27 08:12:45,125 WARN  [c.c.n.NetworkServiceImpl]
(API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Network id=204
failed to restart.

2015-03-27 08:12:45,140 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-16:ctx-0b0aa78a job-189235) Complete async job-189235,
jobStatus: FAILED, resultCode: 530,
result:org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to restart network"}

2015-03-27 08:12:45,152 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-16:ctx-0b0aa78a job-189235) Done executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
job-189235

2015-03-27 08:12:45,158 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-16:ctx-0b0aa78a job-189235) Remove job-189235 from job
monitoring





On Thu, Mar 26, 2015 at 7:38 PM, Somesh Naidu <Somesh.Naidu@citrix.com>
wrote:

You might want to do a arpping on the router's IP from one of the guests
and see how many records are returned.

Somesh
CloudPlatform Escalations
Citrix Systems, Inc.

-----Original Message-----
From: Erik Weber [mailto:terbolous@gmail.com]
Sent: Thursday, March 26, 2015 12:54 PM
To: users@cloudstack.apache.org
Subject: Re: Virtual Routers not responding to dns requests

I briefly remember having similar problems at some point, but do not recall
details as version nor the solution.

1) Does it work if you restart dnsmasq on the VR?
2) is the VR able to do outgoing dns requests?
3) anything in syslog/dnsmasq logs?
4) any egress rules in place?


Erik

Den torsdag 26. mars 2015 skrev cs user <acldstkusr@gmail.com> følgende:

Hi All,

We have upgraded from 4.3 to 4.4.2.

After some issues with starting the systemvm's, the virtual routers no
longer responding to dns requests from the vm's which we start (or
existing
ones).

We have disabled the firewall on the virtual routers and ran tcpdump on
them but we can't see any inbound traffic on port 53 (udp or tcp). If we
log onto the virtual routers and dig locally against eth0 and the alias
on
eth0 both of these return fine with the correct IP.

This is using Xenserver 6.1 as the host.

Has anyone come across this before? dhcp lookups appear to be working
fine.
Is there a firewall rule in place on the router vm's (other than
iptables),
similar to the security groups which are applied by Xen, which is
preventing these requests from hitting the routers?

Many thanks for any help.




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message