cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-7832) MySQL deadlock occurred in resetting job_executing_msid of the completed vm worker job causing job queue stuck.
Date Mon, 03 Nov 2014 18:48:34 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194884#comment-14194884
] 

ASF subversion and git services commented on CLOUDSTACK-7832:
-------------------------------------------------------------

Commit ffaabdc13fde0f0f7b2667a483006e2a4b805f63 in cloudstack's branch refs/heads/master from
[~minchen07]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=ffaabdc ]

CLOUDSTACK-7832: Move some job db update and item purge to
    completeAsyncJob transaction to avoid MySQL deadlock.


> MySQL deadlock occurred in resetting job_executing_msid of the completed vm worker job
causing job queue stuck.
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-7832
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7832
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>            Reporter: Min Chen
>            Assignee: Min Chen
>            Priority: Critical
>             Fix For: 4.5.0
>
>
> This is mainly happening for an vm operation involving two VM work jobs consecutively,
for example, destroyVM which involved two steps: stopVM and deleteAllVMSnapshots. In updating
job_executing_msid column of the first VM worker job, the second VM worker job has been scheduled
to run, then we will see such MySQL deadlock exception happening:
> 2014-09-17 03:26:40,538 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-164:job-831168/job-831176)
Double excep
> tion
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@42c4a7ea:
UPDATE async_
> job SET async_job.job_executing_msid=null WHERE async_job.id = 831176
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Due to this exception, the queue item is not purged for the first vm worker job, then
all the following operations for the same VM will get stuck.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message