cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anthony Xu (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CLOUDSTACK-7726) [Automation][XenServer] Failed to Remove LoadBalancer Rule due to callHostPlugin failure
Date Thu, 20 Nov 2014 00:42:34 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Anthony Xu resolved CLOUDSTACK-7726.
------------------------------------
    Resolution: Won't Fix

can't reproduce, please reopen when you see it again

> [Automation][XenServer] Failed to Remove LoadBalancer Rule due to callHostPlugin failure
> ----------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-7726
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7726
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Automation
>    Affects Versions: 4.5.0
>            Reporter: Chandan Purushothama
>            Assignee: Anthony Xu
>            Priority: Critical
>             Fix For: 4.5.0
>
>         Attachments: management-server.zip
>
>
> =======================================================
> CallHost Plugin Failure while trying to remove the Load Balancing Rule:
> =======================================================
> 2014-10-13 19:20:06,408 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-45:ctx-fb2667f4)
Seq 1-122723089845846136: Executing request
> 2014-10-13 19:20:06,410 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-89:ctx-0a5fbdf2)
callHostPlugin failed for cmd: createFileInDomr with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238,
domrip: 169.254.2.250, filecontents: global
> 	log 127.0.0.1:3914   local0 warning
> 	maxconn 4096
> 	maxpipes 1024
> 	chroot /var/lib/haproxy
> 	user haproxy
> 	group haproxy
> 	daemon
> 	 
> defaults
> 	log     global
> 	mode    tcp
> 	option  dontlognull
> 	retries 3
> 	option redispatch
> 	option forwardfor
> 	option forceclose
> 	timeout connect    5000
> 	timeout client     50000
> 	timeout server     50000
> listen stats_on_public 10.220.166.19:8081
> 	mode http
> 	option httpclose
> 	stats enable
> 	stats uri     /admin?stats
> 	stats realm   Haproxy\ Statistics
> 	stats auth    admin1:AdMiN123
> 	 
> listen 10_220_166_28-2222 10.220.166.28:2222
> 	balance roundrobin
> 	 
> 	 
> ,  due to There was a failure communicating with the plugin.
> 2014-10-13 19:20:06,410 WARN  [c.c.a.m.DirectAgentAttache] (DirectAgent-89:ctx-0a5fbdf2)
Seq 1-122723089845846135: Throwable caught while executing command
> com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: createFileInDomr
with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250, filecontents:
global
> 	log 127.0.0.1:3914   local0 warning
> 	maxconn 4096
> 	maxpipes 1024
> 	chroot /var/lib/haproxy
> 	user haproxy
> 	group haproxy
> 	daemon
> 	 
> defaults
> 	log     global
> 	mode    tcp
> 	option  dontlognull
> 	retries 3
> 	option redispatch
> 	option forwardfor
> 	option forceclose
> 	timeout connect    5000
> 	timeout client     50000
> 	timeout server     50000
> listen stats_on_public 10.220.166.19:8081
> 	mode http
> 	option httpclose
> 	stats enable
> 	stats uri     /admin?stats
> 	stats realm   Haproxy\ Statistics
> 	stats auth    admin1:AdMiN123
> 	 
> listen 10_220_166_28-2222 10.220.166.28:2222
> 	balance roundrobin
> 	 
> 	 
> ,  due to There was a failure communicating with the plugin.
> 	at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3670)
> 	at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createFileInVR(CitrixResourceBase.java:575)
> 	at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:161)
> 	at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:155)
> 	at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:179)
> 	at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:125)
> 	at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:430)
> 	at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64)
> 	at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> 	at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> 	at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,411 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-89:ctx-0a5fbdf2)
Seq 1-122723089845846135: Response Received: 
> 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135:
Processing:  { Ans: , MgmtId: 59408290029589, via: 1, Ver: v1, Flags: 0, [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
callHostPlugin failed for cmd: createFileInDomr with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238,
domrip: 169.254.2.250, filecontents: global\n\tlog 127.0.0.1:3914   local0 warning\n\tmaxconn
4096\n\tmaxpipes 1024\n\tchroot /var/lib/haproxy\n\tuser haproxy\n\tgroup haproxy\n\tdaemon\n\t
\ndefaults\n\tlog     global\n\tmode    tcp\n\toption  dontlognull\n\tretries 3\n\toption
redispatch\n\toption forwardfor\n\toption forceclose\n\ttimeout connect    5000\n\ttimeout
client     50000\n\ttimeout server     50000\n\nlisten stats_on_public 10.220.166.19:8081\n\tmode
http\n\toption httpclose\n\tstats enable\n\tstats uri     /admin?stats\n\tstats realm   Haproxy\\
Statistics\n\tstats auth    admin1:AdMiN123\n\n\t \nlisten 10_220_166_28-2222 10.220.166.28:2222\n\tbalance
roundrobin\n\t \n\t \n,  due to There was a failure communicating with the plugin.","wait":0}}]
}
> 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (API-Job-Executor-41:ctx-9ee60aa8 job-84
ctx-803aafba) Seq 1-122723089845846135: Received:  { Ans: , MgmtId: 59408290029589, via: 1,
Ver: v1, Flags: 0, { Answer } }
> 2014-10-13 19:20:06,424 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84 ctx-803aafba) LB Rollback rule id: 11,while removing vmId 4
> 2014-10-13 19:20:06,427 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84 ctx-803aafba) LB Rollback rule id: 11 while removing vm instances
> 2014-10-13 19:20:06,427 WARN  [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84 ctx-803aafba) Unable to apply the load balancer config because resource is unavaliable.
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Unable to apply loadbalancing rules on router 
> 	at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4050)
> 	at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3901)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at $Proxy190.applyLoadBalancingRules(Unknown Source)
> 	at com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:384)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1845)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2433)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1881)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1805)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1377)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at $Proxy138.removeFromLoadBalancer(Unknown Source)
> 	at org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
> 	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> 	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,444 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-23:ctx-238092a2)
Created a vif 825c5404-4bc8-3a35-8961-db3d95bde69d on 1
> 2014-10-13 19:20:06,452 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Unexpected exception while executing org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to remove specified load balancer
rule id for vms [4]
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1413)
> 	at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at $Proxy138.removeFromLoadBalancer(Unknown Source)
> 	at org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
> 	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> 	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Complete async job-84, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to remove specified load balancer rule id for vms [4]"}
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Publish async job-84 complete on message bus
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Wake up jobs related to job- 84
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Update db status for job- 84
> 2014-10-13 19:20:06,458 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Wake up jobs joined with job- 84 and disjoin all subjobs created from job- 84
> 2014-10-13 19:20:06,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8
job-84) Done executing org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
for job-84



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message