ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Chugunov <sergey.chugu...@gmail.com>
Subject Re: IgniteOutOfMemoryException: Not enough memory allocated
Date Thu, 19 Oct 2017 12:39:47 GMT
Hello Pradeep,

I managed to reproduce the same behavior; its root cause is in
your configuration.

It is not obvious but when you add this piece:

           <property name="memoryPolicies">
                <list>
                        <bean
class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
                                <property name="metricsEnabled"
value="true"/>
                        </bean>
                </list>
             </property>

then property *defaultMemoryPolicySize* is ignored and another value is
used: in Ignite2.2 it is 20% of total physical memory on machine (looking
at 27 Gigs reported in stack trace I think it is exactly your case).

To fix your configuration please do the following: get rid of
*defaultMemoryPolicySize*
property and set up MemoryConfiguration like this:

<property name="memoryConfiguration">
         <bean class="org.apache.ignite.configuration.MemoryConfiguration">

            <property name="pageSize" value="16384"/>

            <property name="memoryPolicies">
                <list>
                        <bean
class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
                         <property name="initialSize" value="#{75L * 1024 *
1024 * 1024}"/>
                         <property name="maxSize" value="#{75L * 1024 *
1024 * 1024}"/>
                        <property name="metricsEnabled"
value="true"/>
                        </bean>
                </list>
             </property>
         </bean>
      </property>

Thanks,
Sergey.

On Wed, Oct 18, 2017 at 9:45 PM, pradeepchanumolu <pradeepbaji@gmail.com>
wrote:

> I am hitting the following exception in the Ignite Client when trying to
> load
> data into Ignite Cache. The exception says that the default policy size is
> 27GB but on the server configuration I have set it to much higher value.
> Here is the snippet of the server configuration.
>
> <property name="memoryConfiguration">
>          <bean class="org.apache.ignite.configuration.
> MemoryConfiguration">
>
>              <property name="defaultMemoryPolicySize" value="#{75L * 1024 *
> 1024 * 1024}"/>
>
>             <property name="pageSize" value="16384"/>
>             <property name="memoryPolicies">
>                 <list>
>                         <bean
> class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
>                                 <property name="metricsEnabled"
> value="true"/>
>                         </bean>
>                 </list>
>              </property>
>          </bean>
>       </property>
>
> Can someone explain why I am hitting this exception? I have 10 servers
> running on separate machines with 100GB of Heap memory allocated to each
> server. The data I am loading is just 350GB and I am hitting this
> exception.
>
>
> Caused by: class
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Failed to locally write to cache (all transaction entries will be
> invalidated, however there was a window when entries for this transaction
> were visible to others): GridNearTxLocal
> [mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
> [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=695,
> val=162247185594843443, hasValBytes=true], cacheId=-1015733162,
> txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=695,
> val=162247185594843443,
> hasValBytes=true], cacheId=-1015733162], val=[op=CREATE,
> val=CacheObjectByteArrayImpl [arrLen=288]], prevVal=[op=NOOP, val=null],
> oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1,
> conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null,
> filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtCacheEntry
> [rdrs=[], part=695, super=GridDistributedCacheEntry
> [super=GridCacheMapEntry
> [key=KeyCacheObjectImpl [part=695, val=162247185594843443,
> hasValBytes=true], val=null, startVer=1508548585654, ver=GridCacheVersion
> [topVer=119776747, order=1508548585654, nodeOrder=313], hash=-1519359033,
> extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc
> [locs=[GridCacheMvccCandidate [nodeId=c63d5132-30ba-483a-
> 8f16-bd450c19d70f,
> ver=GridCacheVersion [topVer=119776747, order=1508548585653,
> nodeOrder=313],
> threadId=687, id=39334849, topVer=AffinityTopologyVersion [topVer=449,
> minorTopVer=0], reentry=null,
> otherNodeId=c63d5132-30ba-483a-8f16-bd450c19d70f,
> otherVer=GridCacheVersion
> [topVer=119776747, order=1508548585653, nodeOrder=313],
> mappedDhtNodes=null,
> mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
> [part=695, val=162247185594843443, hasValBytes=true],
> masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_
> implicit=1|dht_local=1|near_local=0|removed=0|read=0,
> prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=1,
> locked=false, nodeId=c63d5132-30ba-483a-8f16-bd450c19d70f,
> locMapped=false,
> expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
> serReadVer=null, xidVer=GridCacheVersion [topVer=119776747,
> order=1508548585653, nodeOrder=313]]], explicitLock=false, dhtVer=null,
> last=false, nearEntries=0, clientFirst=false,
> node=c63d5132-30ba-483a-8f16-bd450c19d70f]], nearLocallyMapped=false,
> colocatedLocallyMapped=true, needCheckBackup=null, hasRemoteLocks=false,
> thread=data-streamer-#494%null%, mappings=IgniteTxMappingsSingleImpl
> [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry
> [key=KeyCacheObjectImpl [part=695, val=162247185594843443,
> hasValBytes=true], cacheId=-1015733162, txKey=IgniteTxKey
> [key=KeyCacheObjectImpl [part=695, val=162247185594843443,
> hasValBytes=true], cacheId=-1015733162], val=[op=CREATE,
> val=CacheObjectByteArrayImpl [arrLen=288]], prevVal=[op=NOOP, val=null],
> oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1,
> conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null,
> filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtCacheEntry
> [rdrs=[], part=695, super=GridDistributedCacheEntry
> [super=GridCacheMapEntry
> [key=KeyCacheObjectImpl [part=695, val=162247185594843443,
> hasValBytes=true], val=null, startVer=1508548585654, ver=GridCacheVersion
> [topVer=119776747, order=1508548585654, nodeOrder=313], hash=-1519359033,
> extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc
> [locs=[GridCacheMvccCandidate [nodeId=c63d5132-30ba-483a-
> 8f16-bd450c19d70f,
> ver=GridCacheVersion [topVer=119776747, order=1508548585653,
> nodeOrder=313],
> threadId=687, id=39334849, topVer=AffinityTopologyVersion [topVer=449,
> minorTopVer=0], reentry=null,
> otherNodeId=c63d5132-30ba-483a-8f16-bd450c19d70f,
> otherVer=GridCacheVersion
> [topVer=119776747, order=1508548585653, nodeOrder=313],
> mappedDhtNodes=null,
> mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
> [part=695, val=162247185594843443, hasValBytes=true],
> masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_
> implicit=1|dht_local=1|near_local=0|removed=0|read=0,
> prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=1,
> locked=false, nodeId=c63d5132-30ba-483a-8f16-bd450c19d70f,
> locMapped=false,
> expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
> serReadVer=null, xidVer=GridCacheVersion [topVer=119776747,
> order=1508548585653, nodeOrder=313]]], explicitLock=false, dhtVer=null,
> last=false, nearEntries=0, clientFirst=false,
> node=c63d5132-30ba-483a-8f16-bd450c19d70f]], super=GridDhtTxLocalAdapter
> [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[],
> explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
> depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true,
> recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion
> [topVer=119776747, order=1508548585653, nodeOrder=313],
> writeVer=GridCacheVersion [topVer=119776747, order=1508548585655,
> nodeOrder=313], implicit=true, loc=true, threadId=687,
> startTime=1508351793883, nodeId=c63d5132-30ba-483a-8f16-bd450c19d70f,
> startVer=GridCacheVersion [topVer=119776747, order=1508548585653,
> nodeOrder=313], endVer=null, isolation=READ_COMMITTED,
> concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2,
> commitVer=GridCacheVersion [topVer=119776747, order=1508548585653,
> nodeOrder=313], finalizing=NONE, invalidParts=null, state=COMMITTING,
> timedOut=false, topVer=AffinityTopologyVersion [topVer=449, minorTopVer=0],
> duration=0ms, onePhaseCommit=true], size=1]]]
>         at
> org.apache.ignite.internal.processors.cache.transactions.
> IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:834)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal.localFinish(GridNearTxLocal.java:3030)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:405)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal$16.apply(GridNearTxLocal.java:3180)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal$16.apply(GridNearTxLocal.java:3172)
>         at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(
> GridFutureAdapter.java:382)
>         at
> org.apache.ignite.internal.util.future.GridFutureAdapter.
> listen(GridFutureAdapter.java:352)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3172)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal.optimisticPutFuture(GridNearTxLocal.java:2382)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal.putAsync0(GridNearTxLocal.java:623)
>         at
> org.apache.ignite.internal.processors.cache.distributed.
> near.GridNearTxLocal.putAsync(GridNearTxLocal.java:386)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$21.op(
> GridCacheAdapter.java:2355)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$21.op(
> GridCacheAdapter.java:2353)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(
> GridCacheAdapter.java:4055)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(
> GridCacheAdapter.java:2353)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(
> GridCacheAdapter.java:2334)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(
> GridCacheAdapter.java:2311)
>         at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(
> IgniteCacheProxy.java:1502)
>         ... 13 more
> Caused by: class org.apache.ignite.IgniteException: Runtime failure on
> search row:
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$
> SearchRow@7abe4a0d
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.invoke(BPlusTree.java:1632)
>         at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$
> CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1204)
>         at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.
> invoke(IgniteCacheOffheapManagerImpl.java:352)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(
> GridCacheMapEntry.java:3179)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(
> GridCacheMapEntry.java:979)
>         at
> org.apache.ignite.internal.processors.cache.transactions.
> IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:666)
>         ... 30 more
> Caused by: class org.apache.ignite.internal.mem.
> IgniteOutOfMemoryException:
> Not enough memory allocated (consider increasing memory policy size or
> enabling evictions) [policyName=default, size=27.0 GB]
>         at
> org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:292)
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:451)
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> freelist.FreeListImpl.insertDataRow(FreeListImpl.java:489)
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> RowStore.addRow(RowStore.java:84)
>         at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$
> CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1258)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$
> UpdateClosure.call(GridCacheMapEntry.java:3875)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$
> UpdateClosure.call(GridCacheMapEntry.java:3833)
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:2988)
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2882)
>         at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.invokeDown(BPlusTree.java:1713)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Mime
View raw message