Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Wed, 26 Jun 2013 04:51:19 +0000 (UTC)
From: "Abhinandan Prateek (JIRA)" <jira@apache.org>
To: cloudstack-issues@incubator.apache.org
Message-ID: <JIRA.12644025.1366675481692.174614.1372222279924@arcas>
In-Reply-To: <JIRA.12644025.1366675481692@arcas>
References: <JIRA.12644025.1366675481692@arcas>
Subject: [jira] [Assigned] (CLOUDSTACK-2141) During HA process ,  dead lock
 is detected - Caused by:
 com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
 found when trying to get lock; try restarting transaction"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/CLOUDSTACK-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhinandan Prateek reassigned CLOUDSTACK-2141:
----------------------------------------------

    Assignee: Koushik Das
    
> During HA process ,  dead lock is detected - Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction"
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-2141
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2141
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.2.0
>         Environment: build from master
>            Reporter: Sangeetha Hariharan
>            Assignee: Koushik Das
>            Priority: Blocker
>             Fix For: 4.2.0
>
>         Attachments: management-server.rar
>
>
> During HA process ,  dead lock is detected - Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction"
> Following are the steps that was followed:
> Set up - Advanced zone with 3 hosts in a cluster.
> One of the host is already shutdown.
> Shutdown another  Host that has Vms that  is ha enabled ( vm in my case "anan-ha1")
> As part of performing HA on this Vm , I see couple of exceptions being thrown:
> 2013-04-22 14:59:05,738 ERROR [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-52) Failed to start instance VM[User|anan-ha1]
> com.cloud.exception.ConcurrentOperationException: Work steps have been changed: ItWork[69587fc9-2bd1-4183-b89c-216ecdf3be5a-Starting-54-Done]
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:770)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:592)
>         at com.cloud.ha.HighAvailabilityManagerImpl.restart(HighAvailabilityManagerImpl.java:533)
>         at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:831)
> 2013-04-22 14:59:05,743 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-52) Cleaning up resources for the vm VM[User|anan-ha1] in Starting state
> ........................
> 2013-04-22 15:00:15,615 WARN  [agent.manager.DirectAgentAttache] (DirectAgent-317:null) Seq 8-1217527861: Exception Caught while executing command
> com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.223.59.3 to 10.223.59.2 due to com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.223.59.3 to 10.223.59.2after 30 retry
>         at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:443)
>         at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:661)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5583)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2286)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:518)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-04-22 15:00:15,616 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-317:null) Seq 8-1217527861: Response Received:
> 2013-04-22 15:00:15,616 DEBUG [agent.transport.Request] (DirectAgent-317:null) Seq 8-1217527861: Processing:  { Ans: , MgmtId: 7508777239729, via: 8, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.223.59.3 to 10.223.59.2 due to com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.223.59.3 to 10.223.59.2after 30 retry","wait":0}}] }
> 2013-04-22 15:00:15,616 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 8-1217527861: Received:  { Ans: , MgmtId: 7508777239729, via: 8, Ver: v1, Flags: 10, { Answer } }
> 2013-04-22 15:00:15,616 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Details from executing class com.cloud.agent.api.GetVncPortCommand: com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.223.59.3 to 10.223.59.2 due to com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.223.59.3 to 10.223.59.2after 30 retry
> 2013-04-22 15:00:15,617 WARN  [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Caught:
> java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.GetVncPortAnswer
>         at com.cloud.consoleproxy.AgentHookBase.onConsoleAccessAuthentication(AgentHookBase.java:169)
>         at com.cloud.consoleproxy.ConsoleProxyListener.processControlCommand(ConsoleProxyListener.java:61)
>         at com.cloud.agent.manager.AgentManagerImpl.handleControlCommand(AgentManagerImpl.java:348)
>         at com.cloud.agent.manager.AgentManagerImpl.access$200(AgentManagerImpl.java:145)
>         at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1286)
>         at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
>         at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:659)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> HA on this Vm did succeed eventually.
> Attaching management server logs .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira