Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2910D200CFE for ; Thu, 24 Aug 2017 14:10:08 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 2627B16A914; Thu, 24 Aug 2017 12:10:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4BB0516A916 for ; Thu, 24 Aug 2017 14:10:07 +0200 (CEST) Received: (qmail 32260 invoked by uid 500); 24 Aug 2017 12:10:05 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 32239 invoked by uid 99); 24 Aug 2017 12:10:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Aug 2017 12:10:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 38450C0334 for ; Thu, 24 Aug 2017 12:10:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id y4wtBpaveahp for ; Thu, 24 Aug 2017 12:10:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 3477260CE6 for ; Thu, 24 Aug 2017 12:10:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 07259E0D57 for ; Thu, 24 Aug 2017 12:10:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 575B025385 for ; Thu, 24 Aug 2017 12:10:00 +0000 (UTC) Date: Thu, 24 Aug 2017 12:10:00 +0000 (UTC) From: "Ivan Rakov (JIRA)" To: dev@ignite.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (IGNITE-6176) Request to BinaryMetadataTransport may cause deadlock on grid stop MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 24 Aug 2017 12:10:08 -0000 Ivan Rakov created IGNITE-6176: ---------------------------------- Summary: Request to BinaryMetadataTransport may cause deadlock on grid stop Key: IGNITE-6176 URL: https://issues.apache.org/jira/browse/IGNITE-6176 Project: Ignite Issue Type: Bug Affects Versions: 2.1 Reporter: Ivan Rakov Fix For: 2.2 We may access BinaryMetadataTransport while holding striped GridCacheIOManager lock in some cases (for example, during partition exhange). When grid is stopping, BinaryMetadataTransport#requestMetadataUpdate may hang on future. This will result a deadlock: this future won't be cancelled. as we firstly stop GridCacheIOManager (which, in turn, requires releasin striped lock). Steps to reproduce: 1) Remove partition exchange messages from classnames.properties 2) Run IgniteClusterActivateDeactivateTestWithPersistence multiple times Stacktrace of deadlocked threads: {noformat} "sys-#9065%cache.IgniteClusterActivateDeactivateTestWithPersistence4%@18095" prio=5 tid=0x2b55 nid=NA waiting java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Unsafe.java:-1) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.addMeta(CacheObjectBinaryProcessorImpl.java:432) at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.addMeta(CacheObjectBinaryProcessorImpl.java:173) at org.apache.ignite.internal.binary.BinaryContext.updateMetadata(BinaryContext.java:1276) at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:783) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:206) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134) at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteObject(BinaryWriterExImpl.java:496) at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteMap(BinaryWriterExImpl.java:786) at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:699) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:206) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134) at org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:251) at org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58) at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9849) at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9913) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsSingleMessage.prepareMarshal(GridDhtPartitionsSingleMessage.java:289) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onSend(GridCacheIoManager.java:1120) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1154) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSinglePartitionRequest(GridDhtPartitionsExchangeFuture.java:2588) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$1500(GridDhtPartitionsExchangeFuture.java:114) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$6.apply(GridDhtPartitionsExchangeFuture.java:2492) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$6.apply(GridDhtPartitionsExchangeFuture.java:2490) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382) at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:352) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceivePartitionRequest(GridDhtPartitionsExchangeFuture.java:2490) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionRequest(GridCachePartitionExchangeManager.java:1474) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:129) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$4.onMessage(GridCachePartitionExchangeManager.java:337) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$4.onMessage(GridCachePartitionExchangeManager.java:335) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:2574) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:2553) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "test-runner-#8647%cache.IgniteClusterActivateDeactivateTestWithPersistence%@17513" prio=5 tid=0x2956 nid=NA waiting java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Unsafe.java:-1) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115) at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:201) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.writeLock(GridCacheIoManager.java:509) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onKernalStop0(GridCacheIoManager.java:543) at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:120) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:901) at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2149) at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2097) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2463) - locked <0x4d91> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2426) at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:325) at org.apache.ignite.Ignition.stop(Ignition.java:224) at org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:1025) at org.apache.ignite.internal.processors.cache.IgniteClusterActivateDeactivateTest.stateChangeFailover2(IgniteClusterActivateDeactivateTest.java:1112) at org.apache.ignite.internal.processors.cache.IgniteClusterActivateDeactivateTest.testActivateFailover2(IgniteClusterActivateDeactivateTest.java:1070) at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at junit.framework.TestCase.runTest(TestCase.java:176) at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000) at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132) at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915) at java.lang.Thread.run(Thread.java:745) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)