Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BB301200D5F for ; Mon, 18 Dec 2017 15:05:44 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B97CD160C05; Mon, 18 Dec 2017 14:05:44 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3655E160BF9 for ; Mon, 18 Dec 2017 15:05:43 +0100 (CET) Received: (qmail 63708 invoked by uid 500); 18 Dec 2017 14:05:42 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 63696 invoked by uid 99); 18 Dec 2017 14:05:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Dec 2017 14:05:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 798D41A0BB8 for ; Mon, 18 Dec 2017 14:05:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.674 X-Spam-Level: *** X-Spam-Status: No, score=3.674 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, FORGED_YAHOO_RCVD=1.022, KAM_ASCII_DIVIDERS=0.8, NML_ADSP_CUSTOM_MED=1.2, SPF_HELO_PASS=-0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id PcZbuvqgxA8L for ; Mon, 18 Dec 2017 14:05:36 +0000 (UTC) Received: from n6.nabble.com (n6.nabble.com [162.255.23.37]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 7A8A75F39E for ; Mon, 18 Dec 2017 14:05:36 +0000 (UTC) Received: from n6.nabble.com (localhost [127.0.0.1]) by n6.nabble.com (Postfix) with ESMTP id 3042F38B8AD4 for ; Mon, 18 Dec 2017 07:05:35 -0700 (MST) Date: Mon, 18 Dec 2017 07:05:35 -0700 (MST) From: zbyszek To: user@ignite.apache.org Message-ID: <1513605935195-0.post@n6.nabble.com> Subject: Memory leak in GridCachePartitionExchangeManager? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit archived-at: Mon, 18 Dec 2017 14:05:44 -0000 Dear all, I was wondering if this is a know issue which has a chance to be fixed in future or (I hope) it is me who missed something obvious in working with Ignite caches. I have a simple single node test app (built to investigate a memory leak observed in our PROD deployment), that creates c.a. 20 LOCAL caches per sec. with the config below: private IgniteCache createLocalCache(String name) { CacheConfiguration cCfg = new CacheConfiguration<>(); cCfg.setName(name); cCfg.setGroupName("localCaches"); // without group leak is much bigger! cCfg.setStoreKeepBinary(true); cCfg.setCacheMode(CacheMode.LOCAL); cCfg.setOnheapCacheEnabled(false); cCfg.setCopyOnRead(false); cCfg.setBackups(0); cCfg.setWriteBehindEnabled(false); cCfg.setReadThrough(false); cCfg.setReadFromBackup(false); cCfg.setQueryEntities(); return ignite.createCache(cCfg).withKeepBinary(); } The caches are placed in the queue and are picked up by the worker thread which just destorys them after removing from the queue. This setup seems to generate a memory leak of about 1GB per day. When looking at heapdump, I see all space is occupied by instances of java.util.concurrent.ConcurrentSkipListMap$Node: (please copy paste table into notepad to see tables correctly) Objects by class +------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+---------------------------+ | Class | Objects | Shallow Size | Retained Size | +------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+---------------------------+ | java.util.concurrent.ConcurrentSkipListMap$Node | 4,987,415 13 % | 119,697,960 10 % | ~ 1,204,893,605 100 % | | org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask | 4,985,687 13 % | 239,312,976 20 % | ~ 917,361,000 76 % | | org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner | 4,985,680 13 % | 119,656,320 10 % | ~ 558,390,752 46 % | | org.jsr166.ConcurrentHashMap8 | 4,990,926 13 % | 199,637,040 17 % | ~ 439,459,352 36 % | | org.jsr166.LongAdder8 | 4,992,416 13 % | 159,757,312 13 % | ~ 159,757,312 13 % | | org.apache.ignite.lang.IgniteUuid | 4,989,306 13 % | 119,743,344 10 % | ~ 119,745,456 10 % | | java.util.concurrent.ConcurrentSkipListMap$Index | 2,488,987 7 % | 59,735,688 5 % | ~ 119,502,384 10 % | | java.util.concurrent.ConcurrentSkipListMap$HeadIndex | 49 0 % | 1,568 0 % | ~ 106,991,832 9 % | | org.jsr166.ConcurrentHashMap8$ValuesView | 4,985,368 13 % | 79,765,888 7 % | ~ 79,765,888 7 % | | java.util.HashMap$Node | 44,335 0 % | 1,418,720 0 % | ~ 79,618,104 7 % | | java.util.HashMap$Node[] | 13,093 0 % | 1,098,856 0 % | ~ 68,150,520 6 % | | java.util.HashMap | 13,550 0 % | 650,400 0 % | ~ 67,636,112 6 % | | java.util.concurrent.ConcurrentSkipListMap | 10 0 % | 480 0 % | ~ 59,830,768 5 % | Merged paths to java.util.concurrent.ConcurrentSkipListMap$Node instances (first 5 levels) reports no obvious dominator (at least no dominator from my test namespace): Merged paths +--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+ | Name | Objects | Retained Size | Dominators | +--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+ | +--- | 4,987,415 100 % | 1,037,112,344 100 % | | | | | | | | | +--- | 1,245,699 25 % | 1,037,015,992 99 % | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | | | | | | | | | | | | | | | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | | | | | | | | | | | | | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet | | | | | | | | | | | | | +---org.apache.ignite.thread.IgniteThread | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | 3,741,257 75 % | 777,983,576 75 % | 3,741,257 | | | | | | | | | | +--- | 934,092 19 % | 777,912,544 75 % | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | +---org.apache.ignite.thread.IgniteThread | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | 2,806,816 56 % | 583,674,344 56 % | 2,806,816 | | | | | | | | | | | | +--- | 700,884 14 % | 583,622,824 56 % | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | | | | | | | | | | | | | | | | | +---org.apache.ignite.thread.IgniteThread | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | 2,105,683 42 % | 437,877,768 42 % | 2,105,683 | | | | | | | | | | | | | | +--- | 525,764 11 % | 437,840,192 42 % | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index | | | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | | | | | | | | | | | | | | | | | | | +---org.apache.ignite.thread.IgniteThread | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | 1,579,741 32 % | 328,509,568 32 % | 1,579,741 | | | | | | | | | | | | | | | | +--- | 394,396 8 % | 328,481,960 32 % | | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node | 1,185,216 24 % | 246,468,448 24 % | 1,185,216 | | | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | 129 0 % | 27,608 0 % | 2 | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | 178 0 % | 37,576 0 % | 2 | | | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | 178 0 % | 37,576 0 % | 2 | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | 249 0 % | 51,520 0 % | 2 | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | 249 0 % | 51,520 0 % | 2 | | | | | | | | | | | | +---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache | 249 0 % | 51,520 0 % | 2 | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | 348 0 % | 71,008 0 % | 2 | | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | 348 0 % | 71,008 0 % | 2 | | | | | | | | | | | | +---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache | 348 0 % | 71,008 0 % | 2 | | | | | | | | | | | | +--- | 348 0 % | 71,008 0 % | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator | 1 0 % | 24 0 % | 1 | | | | | | | | | | +---org.apache.ignite.thread.IgniteThread | 1 0 % | 24 0 % | 1 | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex | 458 0 % | 96,304 0 % | 11 | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap | 457 0 % | 96,280 0 % | 10 | | | | | | | | | | | | +---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache | 451 0 % | 96,136 0 % | 2 | | | | | | | | | | | | | | +--- | 451 0 % | 96,136 0 % | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$CacheGroupHolder1 | | | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.CacheGroupContext | | | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCacheAffinityManager | | | | | | | | | | | | | | | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedSet | 4 0 % | 96 0 % | 4 | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager | 4 0 % | 96 0 % | 2 | | | | | | | | | | | | | | +--- | 4 0 % | 96 0 % | | | | | | | | | | | | | +--- | 1 0 % | 24 0 % | | | | | | | | | | | | | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet | | | | | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor | | | | | | | | | | | | | | | | | +---org.apache.ignite.thread.IgniteThread | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager | 1 0 % | 24 0 % | 1 | | | | | | | | | | | | +--- | 1 0 % | 24 0 % | | | | | | | | | | | | | +---java.util.LinkedList$Node | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$1 | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2 | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3 | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$4 | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5 | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker | | | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCacheSharedContext | | | | | | | | | | | | | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedMap | 1 0 % | 24 0 % | 1 | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager | 1 0 % | 24 0 % | 1 | | | | | | | | | | +--- | 1 0 % | 24 0 % | | | | | | | | | | | +---java.util.LinkedList$Node | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCacheSharedContext | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$2 | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$3 | | | | | | | | | | | | | +---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$DeadlockDetectionListener | | | | | | | | | | | +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator | 1 0 % | 48 0 % | 1 | | | | | | | | +---org.apache.ignite.thread.IgniteThread | 1 0 % | 48 0 % | 1 | +--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+ Also 2 Bigges objects - Dominators are reported as: Objects with biggest retained size +------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | Name | Retained Size | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | +---java.util.concurrent.ConcurrentSkipListMap [Stack Local] size = 4,985,687 | 59,710,656 5 % | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager | 24,527,600 2 % | | | | | | | | +---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeFutureSet | 24,525,728 2 % | | | | | | | | +---java.util.LinkedList size = 1,000 | 24,525,680 2 % | I have also noticed that during the test the most active thread is always "exchange-worker" with allocation reported by visualvm on the level of 200MB/sec at the end of the test. Please let me know if this is a know issue - and if not - if you would be interested in simple repro app. Thank you in advance, zbyszek. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/