cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayapal Reddy Uradi <jayapalreddy.ur...@citrix.com>
Subject Re: Virtual Routers not responding to dns requests
Date Fri, 27 Mar 2015 17:28:12 GMT
While preparing the nic for VM, depending on which network  address space (public/guest) vlan
the isolation_uri is set.
If there is no vlan for the public network, then VR public nic isolation_uri will be untagged.

Thanks,
Jayapal

On 27-Mar-2015, at 9:01 PM, cs user <acldstkusr@gmail.com>
 wrote:

> Just to add, in the database, the  nics table:
> 
> isolation_uri used to be set to "ec2://untagged"
> for vm_type="DomainRouter" of mode Dhcp.
> 
> This is no longer happening however, they are just set as NULL. I don't see
> a way to set this in the networks table however. How does this
> isolation_uri column get set, is it from the database or hard coded in the
> code?
> 
> 
> 
> 
> 
> On Fri, Mar 27, 2015 at 10:02 AM, cs user <acldstkusr@gmail.com> wrote:
> 
>> Hi Again... :-)
>> 
>> So it looks like the vif on the xenserver is not being setup correctly (or
>> has been removed). The following rule is defined on xen host which the
>> broken router is running on:
>> 
>> Chain r-10864-VM (4 references)
>>> 
>>> target     prot opt source               destination
>>> 
>>> RETURN     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-in vif718.0 --physdev-is-bridged
>>> 
>>> RETURN     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-in vif718.1 --physdev-is-bridged
>>> 
>>> ACCEPT     all  --  anywhere             anywhere
>>> 
>>> 
>>>> 
>> The vif that is mentioned above is not present on the host. As below. But
>> on the working router, the vif in the iptable rule does exist.
>> 
>> 
>> On the host, we also see the following in the logs with the vif mentioned:
>> 
>> Mar 26 14:04:31 xen011 script-vif: vif718.0: writing
>> backend/vif/718/0/hotplug-status=connected
>> Mar 26 14:04:31 xen011 scripts-vif: Setting vif718.1 MTU 1500
>> Mar 26 14:04:31 xen011 scripts-vif: Adding vif718.1 to xapi4 with address
>> fe:ff:ff:ff:ff:ff
>> Mar 26 14:04:31 xen011 scripts-vif: Failed to ip link set vif718.1 address
>> fe:ff:ff:ff:ff:ff
>> Mar 26 14:04:31 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[3233] - ['/sbin/ip', 'link', 'set',
>> 'vif718.1', 'down']
>> Mar 26 14:04:31 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[3233] - ['/sbin/ebtables', '-L',
>> 'FORWARD_vif718.1']
>> Mar 26 14:04:31 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[3233] - ['/sbin/ip', 'link', 'set',
>> 'vif718.1', 'up']
>> Mar 26 14:04:31 xen011 script-vif: vif718.1: writing
>> backend/vif/718/1/hotplug-status=connected
>> Mar 26 14:05:12 xen011 script-vif: vif718.1: removing
>> backend/vif/718/1/hotplug-status
>> Mar 26 14:05:12 xen011 script-vif: vif718.1: removing
>> /xapi/718/hotplug/vif/1/hotplug
>> Mar 26 14:05:12 xen011 scripts-vif: vif718.1 has been removed
>> Mar 26 14:05:12 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[4113] - ['/sbin/ip', 'link', 'set',
>> 'vif718.1', 'down']
>> Mar 26 14:05:12 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[4113] - ['/sbin/ebtables', '-L',
>> 'FORWARD_vif718.1']
>> Mar 26 14:05:13 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[4113] - ['/sbin/ip', 'link', 'set',
>> 'vif718.1', 'up']
>> Mar 26 14:05:13 xen011 script-vif: vif718.0: removing
>> backend/vif/718/0/hotplug-status
>> Mar 26 14:05:13 xen011 script-vif: vif718.0: removing
>> /xapi/718/hotplug/vif/0/hotplug
>> Mar 26 14:05:13 xen011 scripts-vif: vif718.0 has been removed
>> Mar 26 14:05:13 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[4156] - ['/sbin/ip', 'link', 'set',
>> 'vif718.0', 'down']
>> Mar 26 14:05:13 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[4156] - ['/sbin/ebtables', '-L',
>> 'FORWARD_vif718.0']
>> Mar 26 14:05:13 xen011 python:
>> /opt/xensource/libexec/setup-vif-rules[4156] - ['/sbin/ip', 'link', 'set',
>> 'vif718.0', 'up']
>> Mar 26 14:05:24 xen011 fe: 4917 (/sbin/ip addr show dev vif718.0) exitted
>> with code 255
>> Mar 26 14:05:25 xen011 fe: 5062 (/sbin/ip addr show dev vif718.1) exitted
>> with code 255
>> 
>> 
>> List of vif's, 718 is missing now however:
>> 
>> 
>>> vif477.0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:150128316 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:163423985 errors:0 dropped:0 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:598157233 (570.4 MiB)  TX bytes:501933888 (478.6 MiB)
>>> 
>>> 
>>>> vif671.0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:38112 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:71566 errors:0 dropped:0 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:2005682 (1.9 MiB)  TX bytes:92870677 (88.5 MiB)
>>> 
>>> 
>>>> vif696.0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:20049 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:49817 errors:0 dropped:0 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:1215219 (1.1 MiB)  TX bytes:62130987 (59.2 MiB)
>>> 
>>> 
>>>> vif703.0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:1459 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:1803 errors:0 dropped:0 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:48244 (47.1 KiB)  TX bytes:213662 (208.6 KiB)
>>> 
>>> 
>>>> vif719.0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:1571 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:75983 errors:0 dropped:2 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:74416 (72.6 KiB)  TX bytes:3710662 (3.5 MiB)
>>> 
>>> 
>>>> vif719.1  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:7982 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:8513 errors:0 dropped:1 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:1349032 (1.2 MiB)  TX bytes:787782 (769.3 KiB)
>>> 
>>> 
>>>> vif720.0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>> 
>>>         UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
>>> 
>>>         RX packets:75 errors:0 dropped:0 overruns:0 frame:0
>>> 
>>>         TX packets:77 errors:0 dropped:0 overruns:0 carrier:0
>>> 
>>>         collisions:0 txqueuelen:32
>>> 
>>>         RX bytes:3404 (3.3 KiB)  TX bytes:5502 (5.3 KiB)
>>> 
>>> 
>> 
>> 
>> 
>> On Fri, Mar 27, 2015 at 8:57 AM, cs user <acldstkusr@gmail.com> wrote:
>> 
>>> Hi Jayapal,
>>> 
>>> Those two parameters are both set to 1.
>>> 
>>> We have a router which has survived the upgrade, and is still able to
>>> receive and return dns requests from instances. We have checked on the
>>> xenserver host and can see the following iptable config:
>>> 
>>> Chain FORWARD (policy ACCEPT)
>>>> target     prot opt source               destination
>>>> BRIDGE-FIREWALL  all  --  anywhere             anywhere
>>>> PHYSDEV match --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth0+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond3+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond0+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth1+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth3+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth6+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond1+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth2+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth5+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth7+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth4+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond2+ --physdev-is-bridged
>>>> DROP       all  --  anywhere             anywhere
>>> 
>>> 
>>> 
>>> However, on a xenserver which is running a router which is not working,
>>> we can see the following:
>>> 
>>> Chain FORWARD (policy ACCEPT)
>>>> target     prot opt source               destination
>>>> BRIDGE-FIREWALL  all  --  anywhere             anywhere
>>>> PHYSDEV match --physdev-is-bridged
>>>> 
>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth1+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth4+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth3+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth7+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond3+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond0+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth6+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond1+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth5+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond2+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth0+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth2+ --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth1 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth4 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth3 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth7 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond3 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth6 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond1 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth5 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond2 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth0 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out eth2 --physdev-is-bridged
>>>> ACCEPT     all  --  anywhere             anywhere            PHYSDEV
>>>> match --physdev-out bond0 --physdev-is-bridged
>>>> DROP       all  --  anywhere             anywhere
>>> 
>>> 
>>> It seems the config is duplicated but without the pluses. I think these
>>> are similar to wildcard entries?
>>> 
>>> Cheers!
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Mar 27, 2015 at 8:46 AM, Jayapal Reddy Uradi <
>>> jayapalreddy.uradi@citrix.com> wrote:
>>> 
>>>> Silly question but Is your xenserver configured bridge mode related
>>>> settings correctly ?
>>>> 
>>>> #xe-switch-network-backend bridge
>>>> #echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
>>>> #echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables
>>>> 
>>>> Thanks,
>>>> Jayapal
>>>> On 27-Mar-2015, at 1:50 PM, cs user <acldstkusr@gmail.com<mailto:
>>>> acldstkusr@gmail.com>> wrote:
>>>> 
>>>> Hi Somesh,
>>>> 
>>>> arping looks good, the correct mac address is displayed and we get a
>>>> unicast reply from the ip address.
>>>> 
>>>> Erik, tried restarting dnsmasq, all looks fine. VR is able to perform
>>>> outgoing dns requests. There is nothing in syslog/dnsmasq logs that I can
>>>> see. No egress rules are in place. The system vm's are able to perform
>>>> dig's against google's dns, but not the virtual router. It seems it is
>>>> being blocked at the xen level.
>>>> 
>>>> We're seeing the below in the logs when restarting a network (either
>>>> ticking clear config or not). This appears to be similar to :
>>>> 
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-7605
>>>> 
>>>> We are using basic zones, some have multiple pods, others don't. We see
>>>> the
>>>> same error in both. The routers come up though and go green, and dnsmasq
>>>> is
>>>> populated with the relevant info. DNS lookups work locally on the router,
>>>> just not remotely. DHCP is working for new machines which get spun up.
>>>> 
>>>> Is there a way to debug this? I've checked the logs on the router
>>>> (cloud.log) can't see any errors in there.
>>>> 
>>>> 2015-03-27 08:12:45,081 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Implementing
>>>> the
>>>> network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] elements and
>>>> resources as a part of network restart
>>>> 
>>>> 2015-03-27 08:12:45,096 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking
>>>> SecurityGroupProvider to implemenet
>>>> Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
>>>> 
>>>> 2015-03-27 08:12:45,103 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking
>>>> VirtualRouter to implemenet
>>>> Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
>>>> 
>>>> 2015-03-27 08:12:45,112 DEBUG
>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is
>>>> acquired
>>>> for network id 204 as a part of router startup in
>>>> 
>>>> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>>>> : Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()]
>>>> 
>>>> 2015-03-27 08:12:45,119 DEBUG
>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Skipping VR
>>>> deployment: Found a running or starting VR in Pod null id=8
>>>> 
>>>> 2015-03-27 08:12:45,120 DEBUG
>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is
>>>> released
>>>> for network id 204 as a part of router startup in
>>>> 
>>>> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>>>> : Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()]
>>>> 
>>>> 2015-03-27 08:12:45,123 WARN  [o.a.c.e.o.NetworkOrchestrator]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Failed to
>>>> implement network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47]
>>>> elements and resources as a part of network restart due to
>>>> 
>>>> java.lang.NullPointerException
>>>> 
>>>>       at
>>>> 
>>>> com.cloud.network.element.VirtualRouterElement.getRouters(VirtualRouterElement.java:952)
>>>> 
>>>>       at
>>>> 
>>>> com.cloud.network.element.VirtualRouterElement.prepareAggregatedExecution(VirtualRouterElement.java:1099)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1090)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
>>>> 
>>>>       at
>>>> 
>>>> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
>>>> 
>>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> 
>>>>       at
>>>> 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> 
>>>>       at
>>>> 
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> 
>>>>       at java.lang.reflect.Method.invoke(Method.java:601)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>> 
>>>>       at
>>>> 
>>>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>> 
>>>>       at
>>>> 
>>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>> 
>>>>       at $Proxy156.restartNetwork(Unknown Source)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
>>>> 
>>>>       at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>>>> 
>>>>       at
>>>> 
>>>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> 
>>>>       at
>>>> 
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> 
>>>>       at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> 
>>>>       at
>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>> 
>>>>       at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>> 
>>>>       at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> 
>>>>       at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> 
>>>>       at java.lang.Thread.run(Thread.java:722)
>>>> 
>>>> 2015-03-27 08:12:45,125 WARN  [c.c.n.NetworkServiceImpl]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Network id=204
>>>> failed to restart.
>>>> 
>>>> 2015-03-27 08:12:45,140 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235) Complete async job-189235,
>>>> jobStatus: FAILED, resultCode: 530,
>>>> 
>>>> result:org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>> to restart network"}
>>>> 
>>>> 2015-03-27 08:12:45,152 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235) Done executing
>>>> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
>>>> job-189235
>>>> 
>>>> 2015-03-27 08:12:45,158 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-16:ctx-0b0aa78a job-189235) Remove job-189235 from job
>>>> monitoring
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Thu, Mar 26, 2015 at 7:38 PM, Somesh Naidu <Somesh.Naidu@citrix.com>
>>>> wrote:
>>>> 
>>>> You might want to do a arpping on the router's IP from one of the guests
>>>> and see how many records are returned.
>>>> 
>>>> Somesh
>>>> CloudPlatform Escalations
>>>> Citrix Systems, Inc.
>>>> 
>>>> -----Original Message-----
>>>> From: Erik Weber [mailto:terbolous@gmail.com]
>>>> Sent: Thursday, March 26, 2015 12:54 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: Virtual Routers not responding to dns requests
>>>> 
>>>> I briefly remember having similar problems at some point, but do not
>>>> recall
>>>> details as version nor the solution.
>>>> 
>>>> 1) Does it work if you restart dnsmasq on the VR?
>>>> 2) is the VR able to do outgoing dns requests?
>>>> 3) anything in syslog/dnsmasq logs?
>>>> 4) any egress rules in place?
>>>> 
>>>> 
>>>> Erik
>>>> 
>>>> Den torsdag 26. mars 2015 skrev cs user <acldstkusr@gmail.com> følgende:
>>>> 
>>>> Hi All,
>>>> 
>>>> We have upgraded from 4.3 to 4.4.2.
>>>> 
>>>> After some issues with starting the systemvm's, the virtual routers no
>>>> longer responding to dns requests from the vm's which we start (or
>>>> existing
>>>> ones).
>>>> 
>>>> We have disabled the firewall on the virtual routers and ran tcpdump on
>>>> them but we can't see any inbound traffic on port 53 (udp or tcp). If we
>>>> log onto the virtual routers and dig locally against eth0 and the alias
>>>> on
>>>> eth0 both of these return fine with the correct IP.
>>>> 
>>>> This is using Xenserver 6.1 as the host.
>>>> 
>>>> Has anyone come across this before? dhcp lookups appear to be working
>>>> fine.
>>>> Is there a firewall rule in place on the router vm's (other than
>>>> iptables),
>>>> similar to the security groups which are applied by Xen, which is
>>>> preventing these requests from hitting the routers?
>>>> 
>>>> Many thanks for any help.
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 


Mime
View raw message