Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 40D22200D5F for ; Mon, 18 Dec 2017 19:37:05 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3F2F7160C2B; Mon, 18 Dec 2017 18:37:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 18C22160BFB for ; Mon, 18 Dec 2017 19:37:03 +0100 (CET) Received: (qmail 77711 invoked by uid 500); 18 Dec 2017 18:37:03 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 77641 invoked by uid 500); 18 Dec 2017 18:37:03 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 77624 invoked by uid 99); 18 Dec 2017 18:37:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Dec 2017 18:37:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A9A0DC1BF9 for ; Mon, 18 Dec 2017 18:37:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.211 X-Spam-Level: X-Spam-Status: No, score=-99.211 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id cq6CXCbAhynM for ; Mon, 18 Dec 2017 18:37:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 207C25F282 for ; Mon, 18 Dec 2017 18:37:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 97451E01A8 for ; Mon, 18 Dec 2017 18:37:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 45049212F8 for ; Mon, 18 Dec 2017 18:37:00 +0000 (UTC) Date: Mon, 18 Dec 2017 18:37:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-9595) Transactions are not getting retried in case of database deadlock errors MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 18 Dec 2017 18:37:05 -0000 [ https://issues.apache.org/jira/browse/CLOUDSTACK-9595?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D16295397#comment-16295397 ]=20 ASF GitHub Bot commented on CLOUDSTACK-9595: -------------------------------------------- rhtyd closed pull request #1762: CLOUDSTACK-9595 Transactions are not getti= ng retried in case of datab=E2=80=A6 URL: https://github.com/apache/cloudstack/pull/1762 =20 =20 =20 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/engine/schema/src/com/cloud/host/dao/HostDaoImpl.java b/engine= /schema/src/com/cloud/host/dao/HostDaoImpl.java index f12a62816fc..6cb09b7c5ba 100644 --- a/engine/schema/src/com/cloud/host/dao/HostDaoImpl.java +++ b/engine/schema/src/com/cloud/host/dao/HostDaoImpl.java @@ -539,7 +539,6 @@ private boolean canOwnCluster(long clusterId) { public List findAndUpdateDirectAgentToLoad(long lastPingSecond= sAfter, Long limit, long managementServerId) { TransactionLegacy txn =3D TransactionLegacy.currentTxn(); =20 - txn.start(); if (s_logger.isDebugEnabled()) { s_logger.debug("Resetting hosts suitable for reconnect"); } @@ -555,6 +554,7 @@ private boolean canOwnCluster(long clusterId) { s_logger.debug("Acquiring hosts for clusters already owned by = this management server"); } List clusters =3D findClustersOwnedByManagementServer(manage= mentServerId); + txn.start(); if (clusters.size() > 0) { // handle clusters already owned by @managementServerId SearchCriteria sc =3D UnmanagedDirectConnectSearch.cre= ate(); diff --git a/server/src/com/cloud/network/IpAddressManagerImpl.java b/serve= r/src/com/cloud/network/IpAddressManagerImpl.java index f3584d1b650..2785df160ec 100644 --- a/server/src/com/cloud/network/IpAddressManagerImpl.java +++ b/server/src/com/cloud/network/IpAddressManagerImpl.java @@ -291,6 +291,8 @@ =20 SearchBuilder AssignIpAddressSearch; SearchBuilder AssignIpAddressFromPodVlanSearch; + private final Object _allocatedLock =3D new Object(); + private final Object _allocatingLock =3D new Object(); =20 static Boolean rulesContinueOnErrFlag =3D true; =20 @@ -759,7 +761,7 @@ public IPAddressVO doInTransaction(TransactionStatus st= atus) throws Insufficient =20 Filter filter =3D new Filter(IPAddressVO.class, "vlanId", = true, 0l, 1l); =20 - List addrs =3D _ipAddressDao.lockRows(sc, fil= ter, true); + List addrs =3D _ipAddressDao.search(sc, filte= r, false); =20 // If all the dedicated IPs of the owner are in use fetch = an IP from the system pool if (addrs.size() =3D=3D 0 && fetchFromDedicatedRange) { @@ -769,7 +771,7 @@ public IPAddressVO doInTransaction(TransactionStatus st= atus) throws Insufficient fetchFromDedicatedRange =3D false; sc.setParameters("vlanId", nonDedicatedVlanDbIds.t= oArray()); errorMessage.append(", vlanId id=3D" + Arrays.toSt= ring(nonDedicatedVlanDbIds.toArray())); - addrs =3D _ipAddressDao.lockRows(sc, filter, true)= ; + addrs =3D _ipAddressDao.search(sc, filter, false); } } =20 @@ -804,24 +806,21 @@ public IPAddressVO doInTransaction(TransactionStatus = status) throws Insufficient addr.setAllocatedInDomainId(owner.getDomainId()); addr.setAllocatedToAccountId(owner.getId()); addr.setSystem(isSystem); + if (displayIp !=3D null) { addr.setDisplay(displayIp); } =20 - if (assign) { - markPublicIpAsAllocated(addr); - } else { - addr.setState(IpAddress.State.Allocating); - } - addr.setState(assign ? IpAddress.State.Allocated : IpAddre= ss.State.Allocating); - if (vlanUse !=3D VlanType.DirectAttached) { addr.setAssociatedWithNetworkId(guestNetworkId); addr.setVpcId(vpcId); } =20 - _ipAddressDao.update(addr.getId(), addr); - + if (assign) { + markPublicIpAsAllocated(addr); + } else { + markPublicIpAsAllocating(addr); + } return addr; } }); @@ -836,35 +835,51 @@ public IPAddressVO doInTransaction(TransactionStatus = status) throws Insufficient @DB @Override public void markPublicIpAsAllocated(final IPAddressVO addr) { - - Transaction.execute(new TransactionCallbackNoReturn() { - @Override - public void doInTransactionWithoutResult(TransactionStatus sta= tus) { - Account owner =3D _accountMgr.getAccount(addr.getAllocated= ToAccountId()); - synchronized (this) { + synchronized (_allocatedLock) { + Transaction.execute(new TransactionCallbackNoReturn() { + @Override + public void doInTransactionWithoutResult(TransactionStatus= status) { + Account owner =3D _accountMgr.getAccount(addr.getAlloc= atedToAccountId()); if (_ipAddressDao.lockRow(addr.getId(), true) !=3D nul= l) { IPAddressVO userIp =3D _ipAddressDao.findById(addr= .getId()); if (userIp.getState() =3D=3D IpAddress.State.Alloc= ating || addr.getState() =3D=3D IpAddress.State.Free) { addr.setState(IpAddress.State.Allocated); - _ipAddressDao.update(addr.getId(), addr); - // Save usage event - if (owner.getAccountId() !=3D Account.ACCOUNT_= ID_SYSTEM) { - VlanVO vlan =3D _vlanDao.findById(addr.get= VlanId()); - String guestType =3D vlan.getVlanType().to= String(); - if (!isIpDedicated(addr)) { - UsageEventUtils.publishUsageEvent(Even= tTypes.EVENT_NET_IP_ASSIGN, owner.getId(), addr.getDataCenterId(), addr.get= Id(), - addr.getAddress().toString(), - addr.isSourceNat(), guestType,= addr.getSystem(), addr.getClass().getName(), addr.getUuid()); - } - if (updateIpResourceCount(addr)) { - _resourceLimitMgr.incrementResourceCou= nt(owner.getId(), ResourceType.public_ip); + if (_ipAddressDao.update(addr.getId(), addr)) = { + // Save usage event + if (owner.getAccountId() !=3D Account.ACCO= UNT_ID_SYSTEM) { + VlanVO vlan =3D _vlanDao.findById(addr= .getVlanId()); + String guestType =3D vlan.getVlanType(= ).toString(); + if (!isIpDedicated(addr)) { + UsageEventUtils.publishUsageEvent(= EventTypes.EVENT_NET_IP_ASSIGN, owner.getId(), addr.getDataCenterId(), addr= .getId(), + addr.getAddress().toString= (), + addr.isSourceNat(), guestT= ype, addr.getSystem(), addr.getClass().getName(), addr.getUuid()); + } + if (updateIpResourceCount(addr)) { + _resourceLimitMgr.incrementResourc= eCount(owner.getId(), ResourceType.public_ip); + } } } } } } - } - }); + }); + } + } + + @DB + private void markPublicIpAsAllocating(final IPAddressVO addr) { + synchronized (_allocatingLock) { + Transaction.execute(new TransactionCallbackNoReturn() { + @Override + public void doInTransactionWithoutResult(TransactionStatus= status) { + + if (_ipAddressDao.lockRow(addr.getId(), true) !=3D nul= l) { + addr.setState(IpAddress.State.Allocating); + _ipAddressDao.update(addr.getId(), addr); + } + } + }); + } } =20 private boolean isIpDedicated(IPAddressVO addr) { @@ -901,40 +916,28 @@ public PublicIp assignDedicateIpAddress(Account owner= , final Long guestNtwkId, f =20 PublicIp ip =3D null; try { - ip =3D Transaction.execute(new TransactionCallbackWithExceptio= n() { - @Override - public PublicIp doInTransaction(TransactionStatus status) = throws InsufficientAddressCapacityException { - Account owner =3D _accountDao.acquireInLockTable(owner= Id); - - if (owner =3D=3D null) { - // this ownerId comes from owner or type Account. = See the class "AccountVO" and the annotations in that class - // to get the table name and field name that is qu= eried to fill this ownerid. - ConcurrentOperationException ex =3D new Concurrent= OperationException("Unable to lock account"); - throw ex; - } - if (s_logger.isDebugEnabled()) { - s_logger.debug("lock account " + ownerId + " is ac= quired"); - } - boolean displayIp =3D true; - if (guestNtwkId !=3D null) { - Network ntwk =3D _networksDao.findById(guestNtwkId= ); - displayIp =3D ntwk.getDisplayNetwork(); - } else if (vpcId !=3D null) { - VpcVO vpc =3D _vpcDao.findById(vpcId); - displayIp =3D vpc.isDisplay(); - } + Account ownerAccount =3D _accountDao.acquireInLockTable(ownerI= d); =20 - PublicIp ip =3D fetchNewPublicIp(dcId, null, null, own= er, VlanType.VirtualNetwork, guestNtwkId, isSourceNat, false, null, false, = vpcId, displayIp); - IPAddressVO publicIp =3D ip.ip(); - - markPublicIpAsAllocated(publicIp); - _ipAddressDao.update(publicIp.getId(), publicIp); + if (ownerAccount =3D=3D null) { + // this ownerId comes from owner or type Account. See the = class "AccountVO" and the annotations in that class + // to get the table name and field name that is queried to= fill this ownerid. + ConcurrentOperationException ex =3D new ConcurrentOperatio= nException("Unable to lock account"); + throw ex; + } + if (s_logger.isDebugEnabled()) { + s_logger.debug("lock account " + ownerId + " is acquired")= ; + } + boolean displayIp =3D true; + if (guestNtwkId !=3D null) { + Network ntwk =3D _networksDao.findById(guestNtwkId); + displayIp =3D ntwk.getDisplayNetwork(); + } else if (vpcId !=3D null) { + VpcVO vpc =3D _vpcDao.findById(vpcId); + displayIp =3D vpc.isDisplay(); + } =20 - return ip; - } - }); + return fetchNewPublicIp(dcId, null, null, owner, VlanType.Virt= ualNetwork, guestNtwkId, isSourceNat, false, null, false, vpcId, displayIp)= ; =20 - return ip; } finally { if (owner !=3D null) { if (s_logger.isDebugEnabled()) { =20 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. =20 For queries about this service, please contact Infrastructure at: users@infra.apache.org > Transactions are not getting retried in case of database deadlock errors > ------------------------------------------------------------------------ > > Key: CLOUDSTACK-9595 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-959= 5 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Affects Versions: 4.8.0 > Reporter: subhash yedugundla > Fix For: 4.8.1 > > > Customer is seeing occasional error 'Deadlock found when trying to get lo= ck; try restarting transaction' messages in their management server logs. = It happens regularly at least once a day. The following is the error seen= =20 > 2015-12-09 19:23:19,450 ERROR [cloud.api.ApiServer] (catalina-exec-3:ctx-= f05c58fc ctx-39c17156 ctx-7becdf6e) unhandled exception executing api comma= nd: [Ljava.lang.String;@230a6e7f > com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mys= ql.jdbc.JDBC4PreparedStatement@74f134e3: DELETE FROM instance_group_vm_map = WHERE instance_group_vm_map.instance_id =3D 941374 > =09at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1209) > =09at sun.reflect.GeneratedMethodAccessor360.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflec= tion(AopUtils.java:317) > =09at org.springframework.aop.framework.ReflectiveMethodInvocation.invoke= Joinpoint(ReflectiveMethodInvocation.java:183) > =09at org.springframework.aop.framework.ReflectiveMethodInvocation.procee= d(ReflectiveMethodInvocation.java:150) > =09at com.cloud.utils.db.TransactionContextInterceptor.invoke(Transaction= ContextInterceptor.java:34) > =09at org.springframework.aop.framework.ReflectiveMethodInvocation.procee= d(ReflectiveMethodInvocation.java:161) > =09at org.springframework.aop.interceptor.ExposeInvocationInterceptor.inv= oke(ExposeInvocationInterceptor.java:91) > =09at org.springframework.aop.framework.ReflectiveMethodInvocation.procee= d(ReflectiveMethodInvocation.java:172) > =09at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDyna= micAopProxy.java:204) > =09at com.sun.proxy.$Proxy237.expunge(Unknown Source) > =09at com.cloud.vm.UserVmManagerImpl$2.doInTransactionWithoutResult(UserV= mManagerImpl.java:2593) > =09at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(Tran= sactionCallbackNoReturn.java:25) > =09at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:5= 7) > =09at com.cloud.utils.db.Transaction.execute(Transaction.java:45) > =09at com.cloud.utils.db.Transaction.execute(Transaction.java:54) > =09at com.cloud.vm.UserVmManagerImpl.addInstanceToGroup(UserVmManagerImpl= .java:2575) > =09at com.cloud.vm.UserVmManagerImpl.updateVirtualMachine(UserVmManagerIm= pl.java:2332) -- This message was sent by Atlassian JIRA (v6.4.14#64029)