cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan X Yue" <ivan...@ca.ibm.com>
Subject Re: Unable to schedule async job
Date Thu, 10 Jan 2019 18:17:36 GMT
Dag,

That's good idea.  How can I update log to debug / trace mode? 

Thanks
Ivan



From:   Dag Sonstebo <Dag.Sonstebo@shapeblue.com>
To:     "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Date:   2019/01/10 11:40 AM
Subject:        Re: Unable to schedule async job



Hi Ivan,

Probably a good idea to bump your logging up to debug or trace – these 
show the SQL queries being prepared in the logs.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue


From: Ivan X Yue <ivanyue@ca.ibm.com>
Reply-To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Date: Thursday, 10 January 2019 at 16:18
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: Re: Unable to schedule async job

Hi, Dag,

Thank for the reply.

Below is my async_job table.  Today, I try to delete some VM. 
Interestingly, I can delete some VM and some are failed withthis error:

Caused by: 
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: 
Duplicate entry '387' for key 'PRIMARY'

But in the async_job table, I don't see id=387 there.

I have attached part of the Management Server log wrt the delete VM 
action.

One minor correction.  I am actually using cloudstack 4.11.1 instead of 
4.9.2.

MariaDB [cloud]> select id, job_cmd from async_job order by id;
+-----+-----------------------------------------------------------------+
| id  | job_cmd                                                         |
+-----+-----------------------------------------------------------------+
| 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
| 276 | NULL                                                            |
| 281 | NULL                                                            |
| 287 | NULL                                                            |
| 288 | NULL                                                            |
| 292 | NULL                                                            |
| 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd   |
| 314 | NULL                                                            |
| 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd   |
| 316 | com.cloud.vm.VmWorkStart                                        |
| 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 321 | NULL                                                            |
| 322 | com.cloud.vm.VmWorkStop                                         |
| 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 327 | com.cloud.vm.VmWorkStart                                        |
| 328 | com.cloud.vm.VmWorkStart                                        |
| 329 | NULL                                                            |
| 330 | com.cloud.vm.VmWorkStop                                         |
| 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 334 | com.cloud.vm.VmWorkStop                                         |
| 335 | com.cloud.vm.VmWorkStart                                        |
| 336 | com.cloud.vm.VmWorkStart                                        |
| 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin |
| 340 | com.cloud.vm.VmWorkStop                                         |
| 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 342 | com.cloud.vm.VmWorkStop                                         |
| 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd        |
| 344 | com.cloud.vm.VmWorkStart                                        |
| 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 346 | com.cloud.vm.VmWorkStop                                         |
| 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
| 348 | com.cloud.vm.VmWorkStop                                         |
| 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd        |
| 350 | com.cloud.vm.VmWorkStart                                        |
| 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 352 | com.cloud.vm.VmWorkStop                                         |
| 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
| 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 356 | com.cloud.vm.VmWorkStop                                         |
| 357 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots                |
| 358 | com.cloud.vm.VmWorkStop                                         |
| 359 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
| 360 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 361 | com.cloud.vm.VmWorkStop                                         |
| 362 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots                |
| 363 | com.cloud.vm.VmWorkStop                                         |
| 364 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 365 | com.cloud.vm.VmWorkStop                                         |
| 366 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots                |
| 367 | com.cloud.vm.VmWorkStop                                         |
| 368 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
| 369 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
| 370 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 372 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 374 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 376 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
| 377 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 378 | com.cloud.vm.VmWorkStop                                         |
| 380 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 381 | com.cloud.vm.VmWorkStop                                         |
| 383 | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin    |
| 384 | com.cloud.vm.VmWorkStart                                        |
| 385 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
| 386 | com.cloud.vm.VmWorkStop                                         |
+-----+-----------------------------------------------------------------+



Thanks
------------------------------------------------------------------------
Ivan Yue
Development Manager - CLM SaaS Operations
Persistent Systems Limited
Email: ivanyue@ca.ibm.com
Email: ivan_yue@persistent.com
------------------------------------------------------------------------



Dag.Sonstebo@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
 
 



From:        Dag Sonstebo <Dag.Sonstebo@shapeblue.com>
To:        "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Date:        2019/01/10 05:01 AM
Subject:        Re: Unable to schedule async job
________________________________



What does your async_job table say around id>330?

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue


On 10/01/2019, 05:34, "Ivan X Yue" <ivanyue@ca.ibm.com> wrote:

   Hi,

   I am using CloudStack 4.9.2 with KVM hypervisors.  Today, I find that 
the
   hypervisor is not responding, and therefore I restart it.  After that, 
I
   find that virtual routers are stopped.  When I try to start them, I 
keep
   getting "Unable to schedule async job" error.

   From the management-server.log, I see some exception related to MySQL:

   Caused by:
 
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
   Duplicate entry '333' for key 'PRIMARY'

   Is there any cleanup that I need to do in the database?  How can I do
   that?


   Here is the full stacktrace of the exception that I get:


   2019-01-09 12:06:31,417 WARN  [o.a.c.f.j.i.AsyncJobManagerImpl]
   (API-Job-Executor-5:ctx-3248f525 job-332 ctx-d797e8bd) (logid:d37b7ec3)
   Unable to schedule async job for command com.cloud.vm.VmWorkStop,
   unexpected exception.
   javax.persistence.EntityExistsException: Entity already exists:
           at
   com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
           at
   com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
           at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
           at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
           at
 
com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
           at
 
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at
 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at
 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at
 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
           at
 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
           at
 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at
 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
           at
 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
           at
 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
           at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source)
           at
 
org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
           at
 
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
           at
 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
           at
 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
           at
 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
           at
 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
           at
 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
           at
   java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at
 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at
 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   Caused by:
 
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
   Duplicate entry '333' for key 'PRIMARY'
           at 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
   Method)
           at
 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
           at
 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
           ... 55 more
   2019-01-09 12:06:31,418 ERROR [c.c.a.ApiAsyncJobDispatcher]
   (API-Job-Executor-5:ctx-3248f525 job-332) (logid:d37b7ec3) Unexpected
   exception while executing
   org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd
   com.cloud.utils.exception.CloudRuntimeException: Unable to schedule 
async
   job for command com.cloud.vm.VmWorkStop, unexpected exception.
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
           at
 
com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
           at
 
com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
           at
 
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at
 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at
 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at
 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
           at
 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
           at
 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at
 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
           at
 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
           at
 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
           at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source)
           at
 
org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
           at
 
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
           at
 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
           at
 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
           at
 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
           at
 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
           at
 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
           at
 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
           at
   java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at
 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at
 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   2019-01-09 12:06:31,419 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
   (API-Job-Executor-5:ctx-3248f525 job-332) (logid:d37b7ec3) Complete 
async
   job-332, jobStatus: FAILED, resultCode: 530, result:
 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
   to schedule async job for command com.cloud.vm.VmWorkStop, unexpected
   exception."}

   Thanks for all the helps in advance.


   Thanks,
   Ivan





Dag.Sonstebo@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue











Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message