ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Semen Boikov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-8917) IgniteOutOfMemory when performing putAll with configured pageEvictionMode
Date Wed, 12 Dec 2018 09:39:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718685#comment-16718685
] 

Semen Boikov commented on IGNITE-8917:
--------------------------------------

For non-atomic caches or when entry is updated on atomic cache backups memory is evicted right
before single entry is updated, and it works more or less reliable (see usages IgniteCacheDatabaseSharedManager.ensureFreeSpace).
But now putAll operation for atomic cache on primary node calls IgniteCacheDatabaseSharedManager.ensureFreeSpace
not  before each entry update, but only once for putAll. It is implemented this way since
putAll first locks all entries and then updates each entry, and it is not safe to call 'ensureFreeSpace'
for each entry while entries locks are held (it can cause deadlock).

I tried to implement some workaround to predict how much space is needed for putAll and evict
this amount of pages before putAll, but it does not work reliable. First I don't see how
to predict how many pages will be needed to update BPlustTree and  free lists, also before
entry is locked we don't know is it will be update or insert + for invoke operation result
is now known in advance. 

I believe if thread-per-partition is implemented (IGNITE-4682) then it won't be necessary
to lock all entries before update, and ensureFreeSpace can be called right before each entry
update.

> IgniteOutOfMemory when performing putAll with configured pageEvictionMode
> -------------------------------------------------------------------------
>
>                 Key: IGNITE-8917
>                 URL: https://issues.apache.org/jira/browse/IGNITE-8917
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache
>    Affects Versions: 2.5
>            Reporter: Denis Mekhanikov
>            Assignee: Semen Boikov
>            Priority: Major
>         Attachments: IgnitePdsPageEvictionBatchTest.java
>
>
> When {{pageEvictionMode}} is configured and {{IgniteCache#putAll(...)}} operations are
performed, then page eviction may not be applied properly. It results in IgniteOutOfMemory
being thrown.
> PFA test, reproducing the problem.
> Exception:
> {noformat}
> Critical system error detected. Will be handled accordingly to configured handler [hnd=class
o.a.i.failure.NoOpFailureHandler, failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.mem.IgniteOutOfMemoryException: Out of memory in data region [name=evicting-data-region,
initSize=256.0 MiB, maxSize=256.0 MiB, persistenceEnabled=false] Try the following:
>  ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
>  ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
>  ^-- Enable eviction or expiration policies]]
>  class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of memory in data
region [name=evicting-data-region, initSize=256.0 MiB, maxSize=256.0 MiB, persistenceEnabled=false]
Try the following:
>  ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
>  ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
>  ^-- Enable eviction or expiration policies
>  at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:305)
>  at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.allocateDataPage(AbstractFreeList.java:464)
>  at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:491)
>  at org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:97)
>  at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1325)
>  at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.update(GridCacheMapEntry.java:4955)
>  at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4750)
>  at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4464)
>  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3083)
>  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2977)
>  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1726)
>  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1703)
>  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1610)
>  at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1272)
>  at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:370)
>  at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1755)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2420)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:812)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:664)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAll0(GridDhtAtomicCache.java:1068)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAll0(GridDhtAtomicCache.java:648)
>  at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2811)
>  at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1147)
>  at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:836)
>  at org.apache.ignite.internal.processors.cache.persistence.db.IgnitePdsPageEvictionBatchTest.testPageEvictionPutAll(IgnitePdsPageEvictionBatchTest.java:57)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at junit.framework.TestCase.runTest(TestCase.java:176)
>  at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2086)
>  at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:140)
>  at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2001)
>  at java.lang.Thread.run(Thread.java:748){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message