ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "javastuff.sam@gmail.com" <javastuff....@gmail.com>
Subject Out-Of-Memory
Date Thu, 15 Jun 2017 23:10:04 GMT
Hi,

We are using Ignite 1.9, OFFHEAP with SWAP disabled. We are creating caches
programmatic and want to use SQL. 
At one instance creating 350 empty cache ran into out-of-memory.  We are
already setting low queue size for delete history
(IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=100)

Log metrics showing 25% free heap memory and 87% free off-heap. Can you
please help me understand this issue and possible resolution? OOM with heap
or offheap or H2 ?

Below is Ignite log with stack trace as well as periodic metrics.

=============
2017-06-15 11:41:41,404,[exchange-worker-#68%TESTNODE%],Skipping rebalancing
(nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
minorTopVer=348], evt=DISCOVERY_CUSTOM_EVT,
node=0c23715a-8c78-4852-9b1a-a5f101d65b7b]
2017-06-15 11:41:43,345,[exchange-worker-#68%TESTNODE%],Failed to
reinitialize local partitions (preloading will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=349], nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT]
class
org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
Failed to allocate memory [total=0, failed=16384]
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(GridUnsafeMemory.java:182)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocateSystem(GridUnsafeMemory.java:145)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(GridUnsafeMap.java:273)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(GridUnsafeMap.java:248)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<init>(GridUnsafePartitionedMap.java:111)
	at
org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.unsafePartitionedMap(GridOffHeapMapFactory.java:224)
	at
org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(GridOffHeapProcessor.java:72)
	at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.initOffHeap(GridCacheSwapManager.java:251)
	at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(GridCacheSwapManager.java:138)
	at
org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(GridCacheManagerAdapter.java:50)
	at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1091)
	at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1745)
	at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1636)
	at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:382)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:581)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:464)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:745)
2017-06-15 11:41:48,111,[grid-timeout-worker-#54%TESTNODE%],
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0c23715a, name=TESTNODE, uptime=00:10:06:422]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=2]
    ^-- CPU [cur=0.27%, avg=20.87%, GC=0%]
    ^-- Heap [used=1461MB, free=26.63%, comm=1991MB]
    ^-- Non heap [used=195MB, free=87.16%, comm=208MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
    ^-- Outbound messages queue [size=0]
2017-06-15 11:41:48,333,[exchange-worker-#68%TESTNODE%],Failed to wait for
completion of partition map exchange (preloading will not start):
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], topVer=1, nodeId8=0c23715a, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1497507101444]], crd=TcpDiscoveryNode
[id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT], added=true,
initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1497507101444,
endTime=1497507107577, ignoreInterrupts=false, state=DONE], init=false,
topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=null,
skipPreload=false, clientOnlyExchange=false, initTs=1497507101444,
centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[],
srvNodes=[TcpDiscoveryNode [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1,
intOrder=1, lastExchangeTime=1497506480276, loc=true,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], super=GridFutureAdapter
[resFlag=1, res=class o.a.i.i.util.offheap.GridOffHeapOutOfMemoryException:
Failed to allocate memory [total=0, failed=16384], startTime=1497507101444,
endTime=1497507107577, ignoreInterrupts=false, state=DONE]]
class org.apache.ignite.IgniteCheckedException: Failed to allocate memory
[total=0, failed=16384]
	at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7239)
	at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:198)
	at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1680)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:745)
Caused by: class
org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
Failed to allocate memory [total=0, failed=16384]
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(GridUnsafeMemory.java:182)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocateSystem(GridUnsafeMemory.java:145)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(GridUnsafeMap.java:273)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(GridUnsafeMap.java:248)
	at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<init>(GridUnsafePartitionedMap.java:111)
	at
org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.unsafePartitionedMap(GridOffHeapMapFactory.java:224)
	at
org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(GridOffHeapProcessor.java:72)
	at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.initOffHeap(GridCacheSwapManager.java:251)
	at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(GridCacheSwapManager.java:138)
	at
org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(GridCacheManagerAdapter.java:50)
	at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1091)
	at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1745)
	at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1636)
	at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:382)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:581)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:464)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
	... 2 more	
	
=============

Thanks,
-Sam



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Out-Of-Memory-tp13829.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message