cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Sonstebo <Dag.Sonst...@shapeblue.com>
Subject Re: Unable to schedule async job
Date Thu, 10 Jan 2019 10:01:05 GMT
What does your async_job table say around id>330?

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
 

On 10/01/2019, 05:34, "Ivan X Yue" <ivanyue@ca.ibm.com> wrote:

    Hi,
    
    I am using CloudStack 4.9.2 with KVM hypervisors.  Today, I find that the 
    hypervisor is not responding, and therefore I restart it.  After that, I 
    find that virtual routers are stopped.  When I try to start them, I keep 
    getting "Unable to schedule async job" error. 
    
    From the management-server.log, I see some exception related to MySQL:
    
    Caused by: 
    com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: 
    Duplicate entry '333' for key 'PRIMARY'
    
    Is there any cleanup that I need to do in the database?  How can I do 
    that? 
    
    
    Here is the full stacktrace of the exception that I get:
    
    
    2019-01-09 12:06:31,417 WARN  [o.a.c.f.j.i.AsyncJobManagerImpl] 
    (API-Job-Executor-5:ctx-3248f525 job-332 ctx-d797e8bd) (logid:d37b7ec3) 
    Unable to schedule async job for command com.cloud.vm.VmWorkStop, 
    unexpected exception.
    javax.persistence.EntityExistsException: Entity already exists:
            at 
    com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
            at 
    com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
            at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
            at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
            at 
    com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
            at 
    com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at 
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at 
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at 
    org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
            at 
    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
            at 
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
            at 
    org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
            at 
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
            at 
    org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
            at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source)
            at 
    org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103)
            at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
            at 
    com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
            at 
    org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
            at 
    org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
            at 
    org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
            at 
    org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
            at 
    org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
            at 
    java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at 
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at 
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
    Caused by: 
    com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: 
    Duplicate entry '333' for key 'PRIMARY'
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
    Method)
            at 
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
            at 
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            ... 55 more
    2019-01-09 12:06:31,418 ERROR [c.c.a.ApiAsyncJobDispatcher] 
    (API-Job-Executor-5:ctx-3248f525 job-332) (logid:d37b7ec3) Unexpected 
    exception while executing 
    org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd
    com.cloud.utils.exception.CloudRuntimeException: Unable to schedule async 
    job for command com.cloud.vm.VmWorkStop, unexpected exception.
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
            at 
    com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
            at 
    com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
            at 
    com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at 
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at 
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at 
    org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
            at 
    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
            at 
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
            at 
    org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
            at 
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
            at 
    org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
            at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source)
            at 
    org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103)
            at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
            at 
    com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
            at 
    org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
            at 
    org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
            at 
    org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
            at 
    org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
            at 
    org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
            at 
    org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
            at 
    java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at 
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at 
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
    2019-01-09 12:06:31,419 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
    (API-Job-Executor-5:ctx-3248f525 job-332) (logid:d37b7ec3) Complete async 
    job-332, jobStatus: FAILED, resultCode: 530, result: 
    org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable

    to schedule async job for command com.cloud.vm.VmWorkStop, unexpected 
    exception."}
    
    Thanks for all the helps in advance.
    
    
    Thanks,
    Ivan
    
    
    


Dag.Sonstebo@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 

Mime
View raw message