ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Cherkasov (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (IGNITE-11400) Rebalancing caches with TTL enabled can cause data corruption.
Date Thu, 21 Mar 2019 15:33:00 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-11400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mikhail Cherkasov resolved IGNITE-11400.
----------------------------------------
    Resolution: Duplicate

> Rebalancing caches with TTL enabled can cause data corruption.
> --------------------------------------------------------------
>
>                 Key: IGNITE-11400
>                 URL: https://issues.apache.org/jira/browse/IGNITE-11400
>             Project: Ignite
>          Issue Type: Bug
>    Affects Versions: 2.5
>            Reporter: Ivan Daschinskiy
>            Assignee: Mikhail Cherkasov
>            Priority: Critical
>
> During or just after rebalancing caches with TTL enabled, data corruption can occurs
while ttl-cleanup-worker purges expired data.
> See details in log
> {code:java}
> [15:24:01,677][INFO ][sys-#49%datafabric-dev-21.example.com%][GridDhtPartitionDemander]
Started rebalance routine [M2_PRODUCT_CACHE, supplier=14c0d3aa-6720-4c7f-a0e5-3ae1a00948b6,
topic=0, fullPartitions=[1, 55, 112, 153, 170, 175, 204, 236, 247, 331, 347, 417, 473, 503,
514, 524, 551, 745, 748, 752, 762, 803, 816, 831, 851, 877, 928, 939], histPartitions=[]]
> [15:24:02,031][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<M2_PRODUCT_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Invalid object type: 0
> 	at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
> 	at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:980)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:299)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:159)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:02,348][INFO ][sys-#52%datafabric-dev-21.example.com%][GridDhtPartitionDemander]
Started rebalance routine [CART_CACHE, supplier=70be2776-9e8f-4940-8a07-5e3c0ad43bdd, topic=0,
fullPartitions=[561, 897], histPartitions=[]]
> [15:24:02,439][INFO ][sys-#48%datafabric-dev-21.example.com%][GridDhtPartitionDemander]
Completed (final) rebalancing [grp=CART_CACHE, supplier=14c0d3aa-6720-4c7f-a0e5-3ae1a00948b6,
topVer=AffinityTopologyVersion [topVer=921, minorTopVer=0], progress=5/5, time=95 ms]
> [15:24:02,558][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<CART_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Item not found: 1
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:02,559][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<M2_PRODUCT_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Invalid object type: 0
> 	at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
> 	at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:980)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:299)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:159)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:03,061][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<CART_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Item not found: 1
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:03,062][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<M2_PRODUCT_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Invalid object type: 0
> 	at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
> 	at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:980)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:299)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:159)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:03,563][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<CART_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Item not found: 1
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:03,564][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<M2_PRODUCT_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Invalid object type: 0
> 	at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
> 	at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:980)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:299)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:159)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:04,066][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<CART_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Item not found: 1
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:04,066][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<M2_PRODUCT_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Invalid object type: 0
> 	at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
> 	at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:980)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:299)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:159)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> [15:24:04,567][ERROR][ttl-cleanup-worker-#39%datafabric-dev-21.example.com%][GridCacheTtlManager]
<CART_CACHE> Failed to process entry expiration: class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on bounds: [lower=null, upper=PendingRow []]
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1000)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:979)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:1957)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:1913)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:861)
> 	at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:207)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:142)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Item not found: 1
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> 	at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> 	at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4702)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4604)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4543)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:956)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:988)
> 	... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message