ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eduard Shangareev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-8768) JVM crash in PDS1 suite in master branch
Date Sun, 10 Jun 2018 10:13:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507330#comment-16507330
] 

Eduard Shangareev commented on IGNITE-8768:
-------------------------------------------

Test which caused JVM Crash - IgniteClusterActivateDeactivateTestWithPersistence#testDeactivateDuringEvictionAndRebalance.

Last interesting logs:
{code}
 [2018-06-08 22:39:56,398][INFO ][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridCacheProcessor]
Stopped cache [cacheName=default]
 [2018-06-08 22:39:56,401][INFO ][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridCacheProcessor]
Stopped cache [cacheName=ignite-sys-cache]
 [2018-06-08 22:39:56,401][ERROR][sys-#23004%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionsEvictor]
Partition eviction failed, this can cause grid hang.
 class org.apache.ignite.IgniteException: Failed to find cache (cache was not started yet
or cache was already stopped): default
 	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.partition(GridCacheAffinityManager.java:176)
 	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.partition(GridCacheAffinityManager.java:160)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.<init>(GridDhtCacheEntry.java:90)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1.create(GridDhtLocalPartition.java:85)
 	at org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:113)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1011)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:859)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:95)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:73)
 	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6710)
 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 	at java.lang.Thread.run(Thread.java:745)
][INFO ][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionDemander]
Cancelled rebalancing from all nodes [topology=AffinityTopologyVersion [topVer=5, minorTopVer=0]]
][INFO ][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionDemander]
Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion
[topVer=5, minorTopVer=0], rebalanceId=1]
][INFO ][db-checkpoint-thread-#22994%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=aab71982-6199-49ad-80e2-e16d36ec14b0, startPtr=FileWALPointer
[idx=0, fileOff=30675421, len=41], checkpointLockWait=0ms, checkpointLockHoldTime=0ms, walCpRecordFsyncDuration=0ms,
pages=120, reason='timeout']
 [2018-06-08 22:39:56,458][ERROR][sys-#23004%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionsEvictor]
Partition eviction failed, this can cause grid hang.
 class org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=null]
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
 	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:1506)
 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.cursor(GridCacheOffheapManager.java:1630)
 	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$6.onHasNext(IgniteCacheOffheapManagerImpl.java:662)
 	at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
 	at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:996)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:859)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:95)
 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:73)
 	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6710)
 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 	at java.lang.Thread.run(Thread.java:745)
 Caused by: class org.apache.ignite.IgniteException: Failed to get page store for the given
cache ID (cache has not been started): 1544803905
 	at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:57)
 	at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:73)
 	at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:146)
 	at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:41)
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4660)
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4562)
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4501)
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:927)
 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:959)
 	... 16 more
 Caused by: class org.apache.ignite.IgniteCheckedException: Failed to get page store for the
given cache ID (cache has not been started): 1544803905
 	at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.getStore(FilePageStoreManager.java:898)
 	at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:361)
 	at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:348)
 	at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
 	at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
 	at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:54)
        [01:39:56]W:		 [org.apache.ignite:ignite-core] 	... 24 more
{code}

Thread which caused crash:
{code}
"sys-#23004%cache.IgniteClusterActivateDeactivateTestWithPersistence2%"
{code}

Its stach trace from core-dump:
{code}
Stack: [0x00007ff2d26e7000,0x00007ff2d27e8000],  sp=0x00007ff2d27e5c70,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 23611 C2 org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(IJZ)J
(1306 bytes) @ 0x00007ff66e599b58 [0x00007ff66e5996c0+0x498]
J 9610 C2 org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(J)J
(104 bytes) @ 0x00007ff66ef2ae48 [0x00007ff66ef2adc0+0x88]
j  org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/ignite/internal/util/lang/GridCursor;+17
J 23614 C1 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/ignite/internal/util/lang/GridCursor;
(196 bytes) @ 0x00007ff6705f048c [0x00007ff6705f0380+0x10c]
J 23023 C2 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/ignite/internal/util/lang/GridCursor;
(8 bytes) @ 0x00007ff66e36af54 [0x00007ff66e36af20+0x34]
j  org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor()Lorg/apache/ignite/internal/util/lang/GridCursor;+6
J 24555 C2 org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$6.onHasNext()Z
(151 bytes) @ 0x00007ff6714fb310 [0x00007ff6714fa6a0+0xc70]
J 7360 C2 org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX()Z (20 bytes)
@ 0x00007ff66e75964c [0x00007ff66e7595e0+0x6c]
j  org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext()Z+1
j  org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll()J+151
j  org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryClear()Z+44
j  org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call()Ljava/lang/Boolean;+186
j  org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call()Ljava/lang/Object;+1
J 20750 C1 org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(Ljava/lang/ClassLoader;Ljava/util/concurrent/Callable;)Ljava/lang/Object;
(57 bytes) @ 0x00007ff66f96731c [0x00007ff66f966680+0xc9c]
J 21885 C1 org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body()V (92
bytes) @ 0x00007ff670824b4c [0x00007ff670824a80+0xcc]
J 20287 C2 org.apache.ignite.internal.util.worker.GridWorker.run()V (1232 bytes) @ 0x00007ff6707de6f0
[0x00007ff6707de500+0x1f0]
J 22515 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(225 bytes) @ 0x00007ff670e03728 [0x00007ff670e034c0+0x268]
J 12836 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007ff66f65bbc4
[0x00007ff66f65bac0+0x104]
J 17745 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007ff66f7b7b28 [0x00007ff66f7b7ae0+0x48]
{code}

> JVM crash in PDS1 suite in master branch
> ----------------------------------------
>
>                 Key: IGNITE-8768
>                 URL: https://issues.apache.org/jira/browse/IGNITE-8768
>             Project: Ignite
>          Issue Type: Bug
>          Components: persistence
>            Reporter: Sergey Chugunov
>            Priority: Blocker
>              Labels: MakeTeamcityGreenAgain
>             Fix For: 2.6
>
>
> JVM crash in latest build: [TC link|https://ci.ignite.apache.org/viewLog.html?buildId=1372456&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_Pds1]
> It is the first crash is latest 15 builds: [TC link|https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Pds1&branch_IgniteTests24Java8=%3Cdefault%3E&tab=buildTypeStatusDiv]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message