cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-9595) Transactions are not getting retried in case of database deadlock errors
Date Fri, 25 Nov 2016 01:03:58 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-9595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694529#comment-15694529
] 

ASF GitHub Bot commented on CLOUDSTACK-9595:
--------------------------------------------

Github user serg38 commented on the issue:

    https://github.com/apache/cloudstack/pull/1762
  
    @rafaelweingartner Looks  like the deadlocks 2 and 3 are the same. I scanned our production
 log  and since last December we had 6400 deadlocks. Out of them close to 6000 were Deadlock
1 
    20 were Deadlock 2 and 700 of a different Deadlock 5. The other deadlocks were in negligible
numbers. I think if we figure out Deadlock 1 and Deadlock 5 this will be good start. I will
try to find the source of transactions for them. In production we run a commercial distribution
based in most part on  4.7 branch of ACS. 
    
    Deadlock 5
    
    *** (1) TRANSACTION:
    TRANSACTION D518886F8, ACTIVE 2 sec fetching rows
    mysql tables in use 4, locked 4
    LOCK WAIT 24 lock struct(s), heap size 3112, 8 row lock(s), undo log entries 17
    MySQL thread id 29781, OS thread handle 0x7f9df36db700, query id 3625404021 ussclpdcsmgt012.autodesk.com
10.41.13.14 cloud Sorting result
    SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address,
user_ip_address.data_center_id, user_ip_address.source_n
    at, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip
    _address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id,
user_ip_address.is_system, user_ip_address.vpc_id, user_ip
    _address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.removed,
user_ip_address.created FROM user_ip_address  INNER JOIN vlan ON user_ip_a
    ddress.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 6  AND user_ip_address.allocated
IS NULL  AND user_ip_address.vlan_db_id IN (32,33,36,37,41,61,62,91,92,93,9
    4,106,107,108,109,11
    *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
    *** (2) TRANSACTION:
    TRANSACTION D5188582B, ACTIVE 17 sec updating or deleting, thread declared inside InnoDB
499
    mysql tables in use 1, locked 1
    25 lock struct(s), heap size 3112, 13 row lock(s), undo log entries 18
    MySQL thread id 29820, OS thread handle 0x7fa35a868700, query id 3625417999 ussclpdcsmgt013.autodesk.com
10.41.13.15 cloud Updating
    UPDATE user_ip_address SET user_ip_address.source_nat=0, user_ip_address.is_system=0,
user_ip_address.account_id=3309, user_ip_address.allocated='2016-03-25 15:36:39', user_ip_address.state='Allocated',
user_ip_address.domain_id=335 WHERE user_ip_address.id = 3284
    *** (2) HOLDS THE LOCK(S):



> Transactions are not getting retried in case of database deadlock errors
> ------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-9595
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9595
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>    Affects Versions: 4.8.0
>            Reporter: subhash yedugundla
>             Fix For: 4.8.1
>
>
> Customer is seeing occasional error 'Deadlock found when trying to get lock; try restarting
transaction' messages in their management server logs.  It happens regularly at least once
a day.  The following is the error seen 
> 2015-12-09 19:23:19,450 ERROR [cloud.api.ApiServer] (catalina-exec-3:ctx-f05c58fc ctx-39c17156
ctx-7becdf6e) unhandled exception executing api command: [Ljava.lang.String;@230a6e7f
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@74f134e3:
DELETE FROM instance_group_vm_map WHERE instance_group_vm_map.instance_id = 941374
> 	at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1209)
> 	at sun.reflect.GeneratedMethodAccessor360.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at com.sun.proxy.$Proxy237.expunge(Unknown Source)
> 	at com.cloud.vm.UserVmManagerImpl$2.doInTransactionWithoutResult(UserVmManagerImpl.java:2593)
> 	at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
> 	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:57)
> 	at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
> 	at com.cloud.utils.db.Transaction.execute(Transaction.java:54)
> 	at com.cloud.vm.UserVmManagerImpl.addInstanceToGroup(UserVmManagerImpl.java:2575)
> 	at com.cloud.vm.UserVmManagerImpl.updateVirtualMachine(UserVmManagerImpl.java:2332)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message