cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jayapal Reddy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-7424) [Automation] Failed to restart network
Date Thu, 28 Aug 2014 05:49:58 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113378#comment-14113378
] 

Jayapal Reddy commented on CLOUDSTACK-7424:
-------------------------------------------

>From the logs, the router is 'stopping' state and network restart operation is performed.
'outer r-229-VM is in Stopping, so not sending apply static nat rules commands to the backend'
2014-08-24 07:31:03,671 WARN  [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Unable to complete shutdown of the network elements due to element:
VpcVirtualRouter
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Unable to remove vm VM[DomainRouter|r-229-VM] from network, is not in the right state

So please update the automation script to check router state before performing network state.
Also please check script operation for what caused the router 'stopping' sate.

Thanks,
Jayapal

> [Automation] Failed to restart network
> --------------------------------------
>
>                 Key: CLOUDSTACK-7424
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7424
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Automation, Test
>    Affects Versions: 4.5.0
>            Reporter: Chandan Purushothama
>            Assignee: Murali Reddy
>            Priority: Critical
>             Fix For: 4.5.0
>
>
> Error Executing the test case *integration.component.test_multiple_ips_per_nic.TestVmNetworkOperations.test_network_restart_cleanup_false_3_VPC*
> *Error Message*
> Job failed: {jobprocstatus : 0, created : u'2014-08-24T07:31:03+0000', cmd : u'org.apache.cloudstack.api.command.user.network.RestartNetworkCmd',
userid : u'77df13d2-2b4d-11e4-89bd-1e5d0e053e75', jobstatus : 2, jobid : u'01faacdc-294b-4209-a855-2dc8ea0aa2b4',
jobresultcode : 530, jobresulttype : u'object', jobresult : {errorcode : 530, errortext :
u'Failed to restart network'}, accountid : u'77df055e-2b4d-11e4-89bd-1e5d0e053e75'}   Logs
available at http://xenrt.hq.xensource.com/control/queue.cgi?action=testlogs&id=804076&phase=Parallel&test=multiple_ips_per_n
> *Stacktrace*
>   File "/usr/lib/python2.7/unittest/case.py", line 332, in run
>     testMethod()
>   File "/usr/local/lib/python2.7/dist-packages/ddt.py", line 114, in wrapper
>     return func(self, *args, **kwargs)
>   File "/root/cloudstack/test/integration/component/test_multiple_ips_per_nic.py", line
1186, in test_network_restart_cleanup_false
>     network.restart(self.apiclient, cleanup=False)
>   File "/usr/local/lib/python2.7/dist-packages/marvin/lib/base.py", line 2626, in restart
>     return(apiclient.restartNetwork(cmd))
>   File "/usr/local/lib/python2.7/dist-packages/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 1899, in restartNetwork
>     response = self.connection.marvinRequest(command, response_type=response, method=method)
>   File "/usr/local/lib/python2.7/dist-packages/marvin/cloudstackConnection.py", line
379, in marvinRequest
>     raise e
> 'Job failed: {jobprocstatus : 0, created : u\'2014-08-24T07:31:03+0000\', cmd : u\'org.apache.cloudstack.api.command.user.network.RestartNetworkCmd\',
userid : u\'77df13d2-2b4d-11e4-89bd-1e5d0e053e75\', jobstatus : 2, jobid : u\'01faacdc-294b-4209-a855-2dc8ea0aa2b4\',
jobresultcode : 530, jobresulttype : u\'object\', jobresult : {errorcode : 530, errortext
: u\'Failed to restart network\'}, accountid : u\'77df055e-2b4d-11e4-89bd-1e5d0e053e75\'}\n
> Logs available at http://xenrt.hq.xensource.com/control/queue.cgi?action=testlogs&id=804076&phase=Parallel&test=multiple_ips_per_n
	
> ==============================
> *Error in the Management Server Log*:
> ==============================
> {code}
> 2014-08-24 07:31:03,602 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Router r-229-VM is in Stopping, so not sending apply static nat rules
commands to the backend
> 2014-08-24 07:31:03,621 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Router r-229-VM is in Stopping, so not sending apply vpc ip association
commands to the backend
> 2014-08-24 07:31:03,638 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Router r-229-VM is in Stopping, so not sending apply vpc ip association
commands to the backend
> 2014-08-24 07:31:03,640 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Sending network shutdown to VpcVirtualRouter
> 2014-08-24 07:31:03,653 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Service SecurityGroup is not supported in the network id=314
> 2014-08-24 07:31:03,654 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Router r-229-VM is in Stopping, so not sending setup guest network
command to the backend
> 2014-08-24 07:31:03,663 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Lock is acquired for nic id 476 as a part of remove vm VM[DomainRouter|r-229-VM]
from network Ntwk[314|Guest|37]
> 2014-08-24 07:31:03,667 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Service SecurityGroup is not supported in the network id=314
> 2014-08-24 07:31:03,668 WARN  [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Unable to remove vm VM[DomainRouter|r-229-VM] from network  Ntwk[314|Guest|37]
> 2014-08-24 07:31:03,669 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Lock is released for nic id 476 as a part of remove vm VM[DomainRouter|r-229-VM]
from network Ntwk[314|Guest|37]
> 2014-08-24 07:31:03,671 WARN  [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Unable to complete shutdown of the network elements due to element:
VpcVirtualRouter
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Unable to remove vm VM[DomainRouter|r-229-VM] from network, is not in the right state
> 	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateRemoveVmFromNetwork(VirtualMachineManagerImpl.java:3079)
> 	at com.cloud.vm.VirtualMachineManagerImpl.removeVmFromNetwork(VirtualMachineManagerImpl.java:3009)
> 	at com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.removeVpcRouterFromGuestNetwork(VpcVirtualNetworkApplianceManagerImpl.java:321)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at $Proxy190.removeVpcRouterFromGuestNetwork(Unknown Source)
> 	at com.cloud.network.element.VpcVirtualRouterElement.shutdown(VpcVirtualRouterElement.java:261)
> 	at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetworkElementsAndResources(NetworkOrchestrator.java:2161)
> 	at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2468)
> 	at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1874)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at $Proxy157.restartNetwork(Unknown Source)
> 	at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
> 	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> 	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 2014-08-24 07:31:03,673 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Failed to shutdown the network elements and resources as a part of
network restart: Implemented
> 2014-08-24 07:31:03,673 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Marking network Ntwk[314|Guest|37] with restartRequired=true
> 2014-08-24 07:31:03,675 WARN  [c.c.n.NetworkServiceImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529 ctx-3eaba8e6) Network id=314 failed to restart.
> 2014-08-24 07:31:03,680 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-82:ctx-6738c61c
job-1529) Complete async job-1529, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to restart network"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message