ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: Out-Of-Memory
Date Fri, 16 Jun 2017 09:14:09 GMT
Hi Sam,

Seems, Ignite can allocate memory for offheap cache map.

Have you tried to reduce CacheConfiguration.startSize property? It is
1_500_000 by default.


On Fri, Jun 16, 2017 at 2:10 AM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:

> Hi,
>
> We are using Ignite 1.9, OFFHEAP with SWAP disabled. We are creating caches
> programmatic and want to use SQL.
> At one instance creating 350 empty cache ran into out-of-memory.  We are
> already setting low queue size for delete history
> (IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=100)
>
> Log metrics showing 25% free heap memory and 87% free off-heap. Can you
> please help me understand this issue and possible resolution? OOM with heap
> or offheap or H2 ?
>
> Below is Ignite log with stack trace as well as periodic metrics.
>
> =============
> 2017-06-15 11:41:41,404,[exchange-worker-#68%TESTNODE%],Skipping
> rebalancing
> (nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
> minorTopVer=348], evt=DISCOVERY_CUSTOM_EVT,
> node=0c23715a-8c78-4852-9b1a-a5f101d65b7b]
> 2017-06-15 11:41:43,345,[exchange-worker-#68%TESTNODE%],Failed to
> reinitialize local partitions (preloading will be stopped):
> GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1,
> minorTopVer=349], nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT]
> class
> org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
> Failed to allocate memory [total=0, failed=16384]
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(
> GridUnsafeMemory.java:182)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.
> allocateSystem(GridUnsafeMemory.java:145)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(
> GridUnsafeMap.java:273)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(
> GridUnsafeMap.java:248)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<
> init>(GridUnsafePartitionedMap.java:111)
>         at
> org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.
> unsafePartitionedMap(GridOffHeapMapFactory.java:224)
>         at
> org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(
> GridOffHeapProcessor.java:72)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.
> initOffHeap(GridCacheSwapManager.java:251)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(
> GridCacheSwapManager.java:138)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(
> GridCacheManagerAdapter.java:50)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(
> GridCacheProcessor.java:1091)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1745)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1636)
>         at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.
> onCacheChangeRequest(CacheAffinitySharedManager.java:382)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onCacheChangeRequest(
> GridDhtPartitionsExchangeFuture.java:581)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur
> e.java:464)
>         at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana
> ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-06-15 11:41:48,111,[grid-timeout-worker-#54%TESTNODE%],
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>     ^-- Node [id=0c23715a, name=TESTNODE, uptime=00:10:06:422]
>     ^-- H/N/C [hosts=1, nodes=1, CPUs=2]
>     ^-- CPU [cur=0.27%, avg=20.87%, GC=0%]
>     ^-- Heap [used=1461MB, free=26.63%, comm=1991MB]
>     ^-- Non heap [used=195MB, free=87.16%, comm=208MB]
>     ^-- Public thread pool [active=0, idle=0, qSize=0]
>     ^-- System thread pool [active=0, idle=6, qSize=0]
>     ^-- Outbound messages queue [size=0]
> 2017-06-15 11:41:48,333,[exchange-worker-#68%TESTNODE%],Failed to wait for
> completion of partition map exchange (preloading will not start):
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
> affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
> super=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
> sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
> lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:
> a8169d0a,
> isClient=false], topVer=1, nodeId8=0c23715a, msg=null,
> type=DISCOVERY_CUSTOM_EVT, tstamp=1497507101444]], crd=TcpDiscoveryNode
> [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
> sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
> lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:
> a8169d0a,
> isClient=false], exchId=GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
> nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT], added=true,
> initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1497507101444,
> endTime=1497507107577, ignoreInterrupts=false, state=DONE], init=false,
> topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=null,
> skipPreload=false, clientOnlyExchange=false, initTs=1497507101444,
> centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[],
> srvNodes=[TcpDiscoveryNode [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b,
> addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1,
> intOrder=1, lastExchangeTime=1497506480276, loc=true,
> ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]],
> super=GridFutureAdapter
> [resFlag=1, res=class o.a.i.i.util.offheap.GridOffHeapOutOfMemoryExceptio
> n:
> Failed to allocate memory [total=0, failed=16384], startTime=1497507101444,
> endTime=1497507107577, ignoreInterrupts=false, state=DONE]]
> class org.apache.ignite.IgniteCheckedException: Failed to allocate memory
> [total=0, failed=16384]
>         at org.apache.ignite.internal.util.IgniteUtils.cast(
> IgniteUtils.java:7239)
>         at
> org.apache.ignite.internal.util.future.GridFutureAdapter.
> get0(GridFutureAdapter.java:198)
>         at
> org.apache.ignite.internal.util.future.GridFutureAdapter.
> get(GridFutureAdapter.java:139)
>         at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana
> ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1680)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: class
> org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
> Failed to allocate memory [total=0, failed=16384]
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(
> GridUnsafeMemory.java:182)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.
> allocateSystem(GridUnsafeMemory.java:145)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(
> GridUnsafeMap.java:273)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(
> GridUnsafeMap.java:248)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<
> init>(GridUnsafePartitionedMap.java:111)
>         at
> org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.
> unsafePartitionedMap(GridOffHeapMapFactory.java:224)
>         at
> org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(
> GridOffHeapProcessor.java:72)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.
> initOffHeap(GridCacheSwapManager.java:251)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(
> GridCacheSwapManager.java:138)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(
> GridCacheManagerAdapter.java:50)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(
> GridCacheProcessor.java:1091)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1745)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1636)
>         at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.
> onCacheChangeRequest(CacheAffinitySharedManager.java:382)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onCacheChangeRequest(
> GridDhtPartitionsExchangeFuture.java:581)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur
> e.java:464)
>         at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana
> ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
>         ... 2 more
>
> =============
>
> Thanks,
> -Sam
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Out-Of-Memory-tp13829.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Best regards,
Andrey V. Mashenkov

Mime
View raw message