Return-Path: <user-return-12559-archive-asf-public=cust-asf.ponee.io@ignite.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 702C2200CA9
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Fri, 16 Jun 2017 11:14:19 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id 6EB4C160BDD; Fri, 16 Jun 2017 09:14:19 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 192D2160BD2
	for <archive-asf-public@cust-asf.ponee.io>; Fri, 16 Jun 2017 11:14:17 +0200 (CEST)
Received: (qmail 2349 invoked by uid 500); 16 Jun 2017 09:14:17 -0000
Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@ignite.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@ignite.apache.org>
List-Post: <mailto:user@ignite.apache.org>
List-Id: <user.ignite.apache.org>
Reply-To: user@ignite.apache.org
Delivered-To: mailing list user@ignite.apache.org
Received: (qmail 2331 invoked by uid 99); 16 Jun 2017 09:14:16 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jun 2017 09:14:16 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C976C1806D5
	for <user@ignite.apache.org>; Fri, 16 Jun 2017 09:14:15 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 4.881
X-Spam-Level: ****
X-Spam-Status: No, score=4.881 tagged_above=-999 required=6.31
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	FREEMAIL_REPLY=1, HTML_MESSAGE=2, KAM_BADIPHTTP=2,
	RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01,
	RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001,
	WEIRD_PORT=0.001] autolearn=disabled
Authentication-Results: spamd3-us-west.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=gmail.com
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
	by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024)
	with ESMTP id 61Nl6-lYmh2a for <user@ignite.apache.org>;
	Fri, 16 Jun 2017 09:14:12 +0000 (UTC)
Received: from mail-ua0-f169.google.com (mail-ua0-f169.google.com [209.85.217.169])
	by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 451445F5A2
	for <user@ignite.apache.org>; Fri, 16 Jun 2017 09:14:11 +0000 (UTC)
Received: by mail-ua0-f169.google.com with SMTP id j53so8607905uaa.2
        for <user@ignite.apache.org>; Fri, 16 Jun 2017 02:14:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to;
        bh=17po7PPjWfQnAWc+2igXvxUlbNXD65KwHjUUYkGzWk4=;
        b=s02GL77BY5+C+Gmp8J+5c26XbrZi0dwJ3Bw7yTjp/2ZOOZNBblSoh9Md5L+fxQFLOy
         HLUBTrgdMquHjzSPqPnq8rswNu3H5p0cvyndf2skV7AqpphWs8A4WkXPZIOViJP9ArOq
         V95JMoFSTo2KroPdSWIsBVzdc19d3fNTkYy96pq/IEfjD7R9HOMVbmPVTBc1jr/2ktv8
         jGlJLCJyju7I9xjjFj/kSmYnIvuyQaVjgwRfIrPrwvmlTKqmC+9gsag0Bz07JuKhjbSZ
         IfxA6+jtDMW4cAayq7dvHAeky7FT6Ye38OGaE99AtexcnHMbMWgHt+1/iaZiKE1SY5mm
         AU1Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:in-reply-to:references:from:date
         :message-id:subject:to;
        bh=17po7PPjWfQnAWc+2igXvxUlbNXD65KwHjUUYkGzWk4=;
        b=d9jbYwuEGJ3X2blWqYcrOMPiir4PUj+wiBnUtGOdC8dj9ov6hQG7/kSwrJ/mYWepPx
         +CeYJ5agpBB9SUiJ+PlHLtq5gbUK5fM+HKGeqmaDcJK614p5lxJuJByzqrre84rQH+/S
         waj4NbMBDm32h0FntoH2Ca76OYqeemLZjYugxcUfuVQeK4gI5kW6UUfp4NiYFI4MNjOg
         VZXSXhgqmCUxdd0Tx/T2NqAIJRl0aekcro3XiX3LFxy1TuLI+0sMFb7aRh0mKuMe8q2b
         Sm5L+GBGoVwDIFGOwKd/SmZm1C3NHf8IlHBOYvlCCvfWYCWxFwXDCwgNvKGPtKsBardX
         qLBA==
X-Gm-Message-State: AKS2vOxmjc3wEqbzr9Ja8sHanZNJuGFgakkMwFDKuEsI93RIJbue73oI
	OUytPxZn+sEE9guOREmByjEo9fKUhQ==
X-Received: by 10.176.82.243 with SMTP id w48mr6497869uaw.150.1497604449958;
 Fri, 16 Jun 2017 02:14:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.159.60.166 with HTTP; Fri, 16 Jun 2017 02:14:09 -0700 (PDT)
In-Reply-To: <1497568204561-13829.post@n6.nabble.com>
References: <1497568204561-13829.post@n6.nabble.com>
From: Andrey Mashenkov <andrey.mashenkov@gmail.com>
Date: Fri, 16 Jun 2017 12:14:09 +0300
Message-ID: <CANCAXEH9S7C9cX11FdbC-NH2-Nrkq-ZshCDuvuJys9oFa0P7wQ@mail.gmail.com>
Subject: Re: Out-Of-Memory
To: user@ignite.apache.org
Content-Type: multipart/alternative; boundary="94eb2c18f7d44101da0552103540"
archived-at: Fri, 16 Jun 2017 09:14:19 -0000

--94eb2c18f7d44101da0552103540
Content-Type: text/plain; charset="UTF-8"

Hi Sam,

Seems, Ignite can allocate memory for offheap cache map.

Have you tried to reduce CacheConfiguration.startSize property? It is
1_500_000 by default.


On Fri, Jun 16, 2017 at 2:10 AM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:

> Hi,
>
> We are using Ignite 1.9, OFFHEAP with SWAP disabled. We are creating caches
> programmatic and want to use SQL.
> At one instance creating 350 empty cache ran into out-of-memory.  We are
> already setting low queue size for delete history
> (IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=100)
>
> Log metrics showing 25% free heap memory and 87% free off-heap. Can you
> please help me understand this issue and possible resolution? OOM with heap
> or offheap or H2 ?
>
> Below is Ignite log with stack trace as well as periodic metrics.
>
> =============
> 2017-06-15 11:41:41,404,[exchange-worker-#68%TESTNODE%],Skipping
> rebalancing
> (nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
> minorTopVer=348], evt=DISCOVERY_CUSTOM_EVT,
> node=0c23715a-8c78-4852-9b1a-a5f101d65b7b]
> 2017-06-15 11:41:43,345,[exchange-worker-#68%TESTNODE%],Failed to
> reinitialize local partitions (preloading will be stopped):
> GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1,
> minorTopVer=349], nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT]
> class
> org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
> Failed to allocate memory [total=0, failed=16384]
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(
> GridUnsafeMemory.java:182)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.
> allocateSystem(GridUnsafeMemory.java:145)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(
> GridUnsafeMap.java:273)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(
> GridUnsafeMap.java:248)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<
> init>(GridUnsafePartitionedMap.java:111)
>         at
> org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.
> unsafePartitionedMap(GridOffHeapMapFactory.java:224)
>         at
> org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(
> GridOffHeapProcessor.java:72)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.
> initOffHeap(GridCacheSwapManager.java:251)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(
> GridCacheSwapManager.java:138)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(
> GridCacheManagerAdapter.java:50)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(
> GridCacheProcessor.java:1091)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1745)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1636)
>         at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.
> onCacheChangeRequest(CacheAffinitySharedManager.java:382)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onCacheChangeRequest(
> GridDhtPartitionsExchangeFuture.java:581)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur
> e.java:464)
>         at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana
> ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-06-15 11:41:48,111,[grid-timeout-worker-#54%TESTNODE%],
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>     ^-- Node [id=0c23715a, name=TESTNODE, uptime=00:10:06:422]
>     ^-- H/N/C [hosts=1, nodes=1, CPUs=2]
>     ^-- CPU [cur=0.27%, avg=20.87%, GC=0%]
>     ^-- Heap [used=1461MB, free=26.63%, comm=1991MB]
>     ^-- Non heap [used=195MB, free=87.16%, comm=208MB]
>     ^-- Public thread pool [active=0, idle=0, qSize=0]
>     ^-- System thread pool [active=0, idle=6, qSize=0]
>     ^-- Outbound messages queue [size=0]
> 2017-06-15 11:41:48,333,[exchange-worker-#68%TESTNODE%],Failed to wait for
> completion of partition map exchange (preloading will not start):
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
> affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
> super=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
> sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
> lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:
> a8169d0a,
> isClient=false], topVer=1, nodeId8=0c23715a, msg=null,
> type=DISCOVERY_CUSTOM_EVT, tstamp=1497507101444]], crd=TcpDiscoveryNode
> [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
> sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
> lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:
> a8169d0a,
> isClient=false], exchId=GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
> nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT], added=true,
> initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1497507101444,
> endTime=1497507107577, ignoreInterrupts=false, state=DONE], init=false,
> topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=null,
> skipPreload=false, clientOnlyExchange=false, initTs=1497507101444,
> centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[],
> srvNodes=[TcpDiscoveryNode [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b,
> addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1,
> intOrder=1, lastExchangeTime=1497506480276, loc=true,
> ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]],
> super=GridFutureAdapter
> [resFlag=1, res=class o.a.i.i.util.offheap.GridOffHeapOutOfMemoryExceptio
> n:
> Failed to allocate memory [total=0, failed=16384], startTime=1497507101444,
> endTime=1497507107577, ignoreInterrupts=false, state=DONE]]
> class org.apache.ignite.IgniteCheckedException: Failed to allocate memory
> [total=0, failed=16384]
>         at org.apache.ignite.internal.util.IgniteUtils.cast(
> IgniteUtils.java:7239)
>         at
> org.apache.ignite.internal.util.future.GridFutureAdapter.
> get0(GridFutureAdapter.java:198)
>         at
> org.apache.ignite.internal.util.future.GridFutureAdapter.
> get(GridFutureAdapter.java:139)
>         at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana
> ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1680)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: class
> org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
> Failed to allocate memory [total=0, failed=16384]
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(
> GridUnsafeMemory.java:182)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.
> allocateSystem(GridUnsafeMemory.java:145)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.
> GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(
> GridUnsafeMap.java:273)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(
> GridUnsafeMap.java:248)
>         at
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<
> init>(GridUnsafePartitionedMap.java:111)
>         at
> org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.
> unsafePartitionedMap(GridOffHeapMapFactory.java:224)
>         at
> org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(
> GridOffHeapProcessor.java:72)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.
> initOffHeap(GridCacheSwapManager.java:251)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(
> GridCacheSwapManager.java:138)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(
> GridCacheManagerAdapter.java:50)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(
> GridCacheProcessor.java:1091)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1745)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.
> prepareCacheStart(GridCacheProcessor.java:1636)
>         at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.
> onCacheChangeRequest(CacheAffinitySharedManager.java:382)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onCacheChangeRequest(
> GridDhtPartitionsExchangeFuture.java:581)
>         at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur
> e.java:464)
>         at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana
> ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
>         ... 2 more
>
> =============
>
> Thanks,
> -Sam
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Out-Of-Memory-tp13829.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Best regards,
Andrey V. Mashenkov

--94eb2c18f7d44101da0552103540
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Sam,<div><br></div><div>Seems, Ignite can allocate memo=
ry for offheap cache map.=C2=A0</div><div><br></div><div>Have you tried to =
reduce CacheConfiguration.startSize property? It is 1_500_000 by default.=
=C2=A0<br><br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmai=
l_quote">On Fri, Jun 16, 2017 at 2:10 AM, <a href=3D"mailto:javastuff.sam@g=
mail.com">javastuff.sam@gmail.com</a> <span dir=3D"ltr">&lt;<a href=3D"mail=
to:javastuff.sam@gmail.com" target=3D"_blank">javastuff.sam@gmail.com</a>&g=
t;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0=
 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We are using Ignite 1.9, OFFHEAP with SWAP disabled. We are creating caches=
<br>
programmatic and want to use SQL.<br>
At one instance creating 350 empty cache ran into out-of-memory.=C2=A0 We a=
re<br>
already setting low queue size for delete history<br>
(IGNITE_ATOMIC_CACHE_DELETE_<wbr>HISTORY_SIZE=3D100)<br>
<br>
Log metrics showing 25% free heap memory and 87% free off-heap. Can you<br>
please help me understand this issue and possible resolution? OOM with heap=
<br>
or offheap or H2 ?<br>
<br>
Below is Ignite log with stack trace as well as periodic metrics.<br>
<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
2017-06-15 11:41:41,404,[exchange-worker-<wbr>#68%TESTNODE%],Skipping rebal=
ancing<br>
(nothing scheduled) [top=3DAffinityTopologyVersion [topVer=3D1,<br>
minorTopVer=3D348], evt=3DDISCOVERY_CUSTOM_EVT,<br>
node=3D0c23715a-8c78-4852-9b1a-<wbr>a5f101d65b7b]<br>
2017-06-15 11:41:43,345,[exchange-worker-<wbr>#68%TESTNODE%],Failed to<br>
reinitialize local partitions (preloading will be stopped):<br>
GridDhtPartitionExchangeId [topVer=3D<wbr>AffinityTopologyVersion [topVer=
=3D1,<br>
minorTopVer=3D349], nodeId=3D0c23715a, evt=3DDISCOVERY_CUSTOM_EVT]<br>
class<br>
org.apache.ignite.internal.<wbr>util.offheap.<wbr>GridOffHeapOutOfMemoryExc=
eptio<wbr>n:<br>
Failed to allocate memory [total=3D0, failed=3D16384]<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMemory.a=
llocate0(<wbr>GridUnsafeMemory.java:182)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMemory.<=
wbr>allocateSystem(<wbr>GridUnsafeMemory.java:145)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap$Segm=
ent.&lt;init&gt;(<wbr>GridUnsafeMap.java:624)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap$Segm=
ent.&lt;init&gt;(<wbr>GridUnsafeMap.java:590)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap.init=
(<wbr>GridUnsafeMap.java:273)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap.&lt;=
init&gt;(<wbr>GridUnsafeMap.java:248)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafePartitio=
nedMap.&lt;<wbr>init&gt;(<wbr>GridUnsafePartitionedMap.java:<wbr>111)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.<wbr>GridOffHeapMapFactory.<wb=
r>unsafePartitionedMap(<wbr>GridOffHeapMapFactory.java:<wbr>224)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.offheap.<wbr>GridOffHeapProcesso=
r.create(<wbr>GridOffHeapProcessor.java:72)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheSwapManager.=
<wbr>initOffHeap(<wbr>GridCacheSwapManager.java:251)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheSwapManager.=
start0(<wbr>GridCacheSwapManager.java:138)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheManagerAdapt=
er.start(<wbr>GridCacheManagerAdapter.java:<wbr>50)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheProcessor.st=
artCache(<wbr>GridCacheProcessor.java:1091)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheProcessor.<w=
br>prepareCacheStart(<wbr>GridCacheProcessor.java:1745)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheProcessor.<w=
br>prepareCacheStart(<wbr>GridCacheProcessor.java:1636)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>CacheAffinitySharedMa=
nager.<wbr>onCacheChangeRequest(<wbr>CacheAffinitySharedManager.<wbr>java:3=
82)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.prelo=
ader.<wbr>GridDhtPartitionsExchangeFutur<wbr>e.onCacheChangeRequest(<wbr>Gr=
idDhtPartitionsExchangeFutur<wbr>e.java:581)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.prelo=
ader.<wbr>GridDhtPartitionsExchangeFutur<wbr>e.init(<wbr>GridDhtPartitionsE=
xchangeFutur<wbr>e.java:464)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCachePartitionExc=
hangeMana<wbr>ger$ExchangeWorker.body(<wbr>GridCachePartitionExchangeMana<w=
br>ger.java:1674)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.worker.GridWorker.run(<wbr>GridWorker.=
java:110)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.<wbr>java:745)<b=
r>
2017-06-15 11:41:48,111,[grid-timeout-<wbr>worker-#54%TESTNODE%],<br>
Metrics for local node (to disable set &#39;metricsLogFrequency&#39; to 0)<=
br>
=C2=A0 =C2=A0 ^-- Node [id=3D0c23715a, name=3DTESTNODE, uptime=3D00:10:06:4=
22]<br>
=C2=A0 =C2=A0 ^-- H/N/C [hosts=3D1, nodes=3D1, CPUs=3D2]<br>
=C2=A0 =C2=A0 ^-- CPU [cur=3D0.27%, avg=3D20.87%, GC=3D0%]<br>
=C2=A0 =C2=A0 ^-- Heap [used=3D1461MB, free=3D26.63%, comm=3D1991MB]<br>
=C2=A0 =C2=A0 ^-- Non heap [used=3D195MB, free=3D87.16%, comm=3D208MB]<br>
=C2=A0 =C2=A0 ^-- Public thread pool [active=3D0, idle=3D0, qSize=3D0]<br>
=C2=A0 =C2=A0 ^-- System thread pool [active=3D0, idle=3D6, qSize=3D0]<br>
=C2=A0 =C2=A0 ^-- Outbound messages queue [size=3D0]<br>
2017-06-15 11:41:48,333,[exchange-worker-<wbr>#68%TESTNODE%],Failed to wait=
 for<br>
completion of partition map exchange (preloading will not start):<br>
GridDhtPartitionsExchangeFutur<wbr>e [dummy=3Dfalse, forcePreload=3Dfalse,<=
br>
reassign=3Dfalse, discoEvt=3DDiscoveryCustomEvent [customMsg=3Dnull,<br>
affTopVer=3D<wbr>AffinityTopologyVersion [topVer=3D1, minorTopVer=3D349],<b=
r>
super=3DDiscoveryEvent [evtNode=3DTcpDiscoveryNode<br>
[id=3D0c23715a-8c78-4852-9b1a-<wbr>a5f101d65b7b, addrs=3D[127.0.0.1],<br>
sockAddrs=3D[/<a href=3D"http://127.0.0.1:49500" rel=3D"noreferrer" target=
=3D"_blank">127.0.0.1:49500</a>], discPort=3D49500, order=3D1, intOrder=3D1=
,<br>
lastExchangeTime=3D<wbr>1497506480276, loc=3Dtrue, ver=3D1.9.0#20170302-sha=
1:<wbr>a8169d0a,<br>
isClient=3Dfalse], topVer=3D1, nodeId8=3D0c23715a, msg=3Dnull,<br>
type=3DDISCOVERY_CUSTOM_EVT, tstamp=3D1497507101444]], crd=3DTcpDiscoveryNo=
de<br>
[id=3D0c23715a-8c78-4852-9b1a-<wbr>a5f101d65b7b, addrs=3D[127.0.0.1],<br>
sockAddrs=3D[/<a href=3D"http://127.0.0.1:49500" rel=3D"noreferrer" target=
=3D"_blank">127.0.0.1:49500</a>], discPort=3D49500, order=3D1, intOrder=3D1=
,<br>
lastExchangeTime=3D<wbr>1497506480276, loc=3Dtrue, ver=3D1.9.0#20170302-sha=
1:<wbr>a8169d0a,<br>
isClient=3Dfalse], exchId=3D<wbr>GridDhtPartitionExchangeId<br>
[topVer=3D<wbr>AffinityTopologyVersion [topVer=3D1, minorTopVer=3D349],<br>
nodeId=3D0c23715a, evt=3DDISCOVERY_CUSTOM_EVT], added=3Dtrue,<br>
initFut=3DGridFutureAdapter [resFlag=3D2, res=3Dfalse, startTime=3D14975071=
01444,<br>
endTime=3D1497507107577, ignoreInterrupts=3Dfalse, state=3DDONE], init=3Dfa=
lse,<br>
topSnapshot=3Dnull, lastVer=3Dnull, partReleaseFut=3Dnull, affChangeMsg=3Dn=
ull,<br>
skipPreload=3Dfalse, clientOnlyExchange=3Dfalse, initTs=3D1497507101444,<br=
>
centralizedAff=3Dfalse, forcedRebFut=3Dnull, evtLatch=3D0, remaining=3D[],<=
br>
srvNodes=3D[TcpDiscoveryNode [id=3D0c23715a-8c78-4852-9b1a-<wbr>a5f101d65b7=
b,<br>
addrs=3D[127.0.0.1], sockAddrs=3D[/<a href=3D"http://127.0.0.1:49500" rel=
=3D"noreferrer" target=3D"_blank">127.0.0.1:49500</a>], discPort=3D49500, o=
rder=3D1,<br>
intOrder=3D1, lastExchangeTime=3D<wbr>1497506480276, loc=3Dtrue,<br>
ver=3D1.9.0#20170302-sha1:<wbr>a8169d0a, isClient=3Dfalse]], super=3DGridFu=
tureAdapter<br>
[resFlag=3D1, res=3Dclass o.a.i.i.util.offheap.<wbr>GridOffHeapOutOfMemoryE=
xceptio<wbr>n:<br>
Failed to allocate memory [total=3D0, failed=3D16384], startTime=3D14975071=
01444,<br>
endTime=3D1497507107577, ignoreInterrupts=3Dfalse, state=3DDONE]]<br>
class org.apache.ignite.<wbr>IgniteCheckedException: Failed to allocate mem=
ory<br>
[total=3D0, failed=3D16384]<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.ignite.internal.<wbr>util.IgniteU=
tils.cast(<wbr>IgniteUtils.java:7239)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.future.GridFutureAdapter.<wbr>get0(Gri=
dFutureAdapter.java:<wbr>198)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.future.GridFutureAdapter.<wbr>get(Grid=
FutureAdapter.java:<wbr>139)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCachePartitionExc=
hangeMana<wbr>ger$ExchangeWorker.body(<wbr>GridCachePartitionExchangeMana<w=
br>ger.java:1680)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.worker.GridWorker.run(<wbr>GridWorker.=
java:110)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.<wbr>java:745)<b=
r>
Caused by: class<br>
org.apache.ignite.internal.<wbr>util.offheap.<wbr>GridOffHeapOutOfMemoryExc=
eptio<wbr>n:<br>
Failed to allocate memory [total=3D0, failed=3D16384]<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMemory.a=
llocate0(<wbr>GridUnsafeMemory.java:182)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMemory.<=
wbr>allocateSystem(<wbr>GridUnsafeMemory.java:145)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap$Segm=
ent.&lt;init&gt;(<wbr>GridUnsafeMap.java:624)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap$Segm=
ent.&lt;init&gt;(<wbr>GridUnsafeMap.java:590)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap.init=
(<wbr>GridUnsafeMap.java:273)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafeMap.&lt;=
init&gt;(<wbr>GridUnsafeMap.java:248)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.unsafe.<wbr>GridUnsafePartitio=
nedMap.&lt;<wbr>init&gt;(<wbr>GridUnsafePartitionedMap.java:<wbr>111)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>util.offheap.<wbr>GridOffHeapMapFactory.<wb=
r>unsafePartitionedMap(<wbr>GridOffHeapMapFactory.java:<wbr>224)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.offheap.<wbr>GridOffHeapProcesso=
r.create(<wbr>GridOffHeapProcessor.java:72)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheSwapManager.=
<wbr>initOffHeap(<wbr>GridCacheSwapManager.java:251)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheSwapManager.=
start0(<wbr>GridCacheSwapManager.java:138)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheManagerAdapt=
er.start(<wbr>GridCacheManagerAdapter.java:<wbr>50)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheProcessor.st=
artCache(<wbr>GridCacheProcessor.java:1091)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheProcessor.<w=
br>prepareCacheStart(<wbr>GridCacheProcessor.java:1745)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheProcessor.<w=
br>prepareCacheStart(<wbr>GridCacheProcessor.java:1636)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>CacheAffinitySharedMa=
nager.<wbr>onCacheChangeRequest(<wbr>CacheAffinitySharedManager.<wbr>java:3=
82)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.prelo=
ader.<wbr>GridDhtPartitionsExchangeFutur<wbr>e.onCacheChangeRequest(<wbr>Gr=
idDhtPartitionsExchangeFutur<wbr>e.java:581)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.prelo=
ader.<wbr>GridDhtPartitionsExchangeFutur<wbr>e.init(<wbr>GridDhtPartitionsE=
xchangeFutur<wbr>e.java:464)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at<br>
org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCachePartitionExc=
hangeMana<wbr>ger$ExchangeWorker.body(<wbr>GridCachePartitionExchangeMana<w=
br>ger.java:1674)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ... 2 more<br>
<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
Thanks,<br>
-Sam<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href=3D"http://apache-ignite-users.70518.x=
6.nabble.com/Out-Of-Memory-tp13829.html" rel=3D"noreferrer" target=3D"_blan=
k">http://apache-ignite-users.<wbr>70518.x6.nabble.com/Out-Of-<wbr>Memory-t=
p13829.html</a><br>
Sent from the Apache Ignite Users mailing list archive at Nabble.com.<br>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><d=
iv>Best regards,<br>Andrey V. Mashenkov<br></div></div></div>
</div>

--94eb2c18f7d44101da0552103540--