Return-Path: X-Original-To: apmail-ignite-user-archive@minotaur.apache.org Delivered-To: apmail-ignite-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E22B419E3F for ; Thu, 7 Apr 2016 12:43:29 +0000 (UTC) Received: (qmail 14174 invoked by uid 500); 7 Apr 2016 12:43:29 -0000 Delivered-To: apmail-ignite-user-archive@ignite.apache.org Received: (qmail 14126 invoked by uid 500); 7 Apr 2016 12:43:29 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 14116 invoked by uid 99); 7 Apr 2016 12:43:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2016 12:43:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 4A3CCC0362 for ; Thu, 7 Apr 2016 12:43:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id NMKTE-xQ1dOw for ; Thu, 7 Apr 2016 12:43:27 +0000 (UTC) Received: from mail-lf0-f43.google.com (mail-lf0-f43.google.com [209.85.215.43]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id A534660E79 for ; Thu, 7 Apr 2016 12:43:26 +0000 (UTC) Received: by mail-lf0-f43.google.com with SMTP id e190so54218631lfe.0 for ; Thu, 07 Apr 2016 05:43:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=Uq87rzVaqBWcADdMbnMhs9Oxc/MPZqXu+be6gMiyRtc=; b=YQp4l1Sr6YmvhbTK3BCcSs8gz/ixQoDEkgkGXN2y9BLxRvb4s7Amod8SDADRHNk4Vu NkkfLEJcEL4DFioN64fkDTFsGr5FYGFissg002RJJiiFivz6YfC/uQsG2K1W7daa5f1m dfWMtxF7Gchk+s+EaOF2pTpo+HPel1tTtwMr2iZvgmAbxw38Usp9fesjdzR0z2evZfuu WSC6yiCoy8kqBX5+VwD3eSlWJfT1WTHRIZFck6YUd88lAKLT51S9iAb8SXkxUfqOsI3g OpB2Rrufgrma8EBQY1cPFlzULkQGhbKG0XdMtRxO/lGaYm4Tb0Yk4t10fg88xTIcrGSZ ww0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=Uq87rzVaqBWcADdMbnMhs9Oxc/MPZqXu+be6gMiyRtc=; b=iTAKdqWBD5xwJltNYIxiPwJS+kmg4IiudtG/J+mWzOOFVDpISO5/BVBnN5iJWZb83e dW6vkqc5wcztppSavNZ+DbnuvH73zZX5mUtmkqO59IfO3A52qFtwnisWpgyDlBir0vgC l8xdCf8Rwhle5tV8u6ALCQIgksZHTJIcYXnZBbuyrT7CV9Nu9JXd3tAUgPOcpGaV16+5 XbLrny+nYxiTM4bC3/DVdrl2x/UOL/JHUyx/PeoKJvUFp2XO30IIA3Jb+Affbo1XiKCG 83kVV6IeN+sUGESP/hS8O7+CQX6tHQrHp+CA8iTLykxTBEbBu/G4GZBp54UB3Nroibc1 63fQ== X-Gm-Message-State: AD7BkJI/HvVtaqNtFwR8CXLpH+eiAes5d92//Fqa1bTMzbuo/JOTqK62biSS9I+uP/A2ffUd29atf1QxafAiQw== MIME-Version: 1.0 X-Received: by 10.25.26.75 with SMTP id a72mr1302728lfa.17.1460032998891; Thu, 07 Apr 2016 05:43:18 -0700 (PDT) Received: by 10.25.215.224 with HTTP; Thu, 7 Apr 2016 05:43:18 -0700 (PDT) In-Reply-To: <5706471F.8050508@gridgain.com> References: <570524AA.5010907@gridgain.com> <5706471F.8050508@gridgain.com> Date: Thu, 7 Apr 2016 15:43:18 +0300 Message-ID: Subject: Re: Unexpected heap usage by GridDhtPartitionTopologyImpl From: Tolga Kavukcu To: user@ignite.apache.org Content-Type: multipart/alternative; boundary=001a1140204a426856052fe46c08 --001a1140204a426856052fe46c08 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Denis, IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter seems like decreased heap usage. I will run longer tests to check heap behaviour. Also i need another help with thread's created by ignite. I found out that ignite creates a cleanup thread named "ttl-cleanup-worker" for each cache. But when cache is destroyed, clean up thread does not deleted. Instead it waits sleeping state at all. My first question is that , is it possible to decrease thread count with a configuration, like "thread pool with x threads" for all caches. Secondly, is "unremoved threads" are expected behaviour. Thanks. On Thu, Apr 7, 2016 at 2:40 PM, Denis Magda wrote: > Hi Tolga, > > GridDhtPartitionTopologyImpl is created per cache. If you destroy a cache > this object should be GCed. However you should use cache.destroy() for th= at. > > Please also make sure that you make "live set" heap dumps only. Try to > perform GC explicitly before making the dump because a collector may clea= n > dead objects much later depending on its heuristics. > > -- > Denis > > On 4/7/2016 8:27 AM, Tolga Kavukcu wrote: > > Hi Denis, > > Thanks for the response. I will try IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SI= ZE parameter. > The screnshots taken from eclipse memory analyser which opens and analyse= s > heap dump. I understand heap requirement for wrapping and indexing off-he= ap > entry positions. But also found out that instances of *org.apache.ignite.= internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl > *is constantly increasing within jvm. > > > I also create and destroy so many small caches during the lifecycle, do > you think that it is possible to destroyed caches leaves a footprint in > heap. > > The previous scrreenshots was dominator tree view of memory analyser. I > attached again with headers. > > You can see that each of GridDhtPartitionTopologyImpl uses 20mb~ heap. > And there are 72 instances of GridDhtPartitionTopologyImpl living. > > Also i attached screenshots of leak suspect report of memory analyzer, > which is taken periodically. You an see that instances of *GridDhtPartiti= onTopologyImpl > keeps increasing. * > > Any ideas or suggestions? > > On Wed, Apr 6, 2016 at 6:00 PM, Denis Magda wrote: > >> Hi Tolga, >> >> GridDhtPartitionTopologyImpl contains list of partitions that belong to = a >> specific node. In case of offheap caches each partition (concurrent map) >> contains set of wrappers around keys->values, stored offheap. The wrappe= r >> holds information that's needed to unswap a value or a key to Java heap >> from offheap when required by a user application. >> So Ignite requires extra space for internal needs even when offheap mode >> is used. >> >> I would recommend you trying to reduce >> IgniteSystemProperties.IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. This is = the >> size of the queue that keeps deleted entries for internal needs as well. >> https://apacheignite.readme.io/v1.5/docs/capacity-planning >> >> BTW, could you explain what columns from your screenshot mean exactly? >> What tool did you use to create the memory snapshot? >> >> -- >> Denis >> >> >> >> On 4/6/2016 3:02 PM, Tolga Kavukcu wrote: >> >> Hi everyone, >> >> I use partitioned ignite cache for a very dynamic data. Means that there >> are many updates,deletes and puts with around 5m~ row. >> >> So to avoid gc pauses i use off-heap mode. But when i analyse heap i see >> that count and heap size of >> *org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPart= itionTopologyImpl* is >> increasing constantly. >> >> Please see attached screenshots taken from mat heap dump. >> >> = >> >> Thanks for helping out. >> >> There are totally 1.2 heap used by GridDhtPartitionTopologyImpl, almost >> equals to my data size. Do you think that there are problems with >> configuration. >> >> >> *Tolga KAVUK=C3=87U * >> >> >> > > > -- > > *Tolga KAVUK=C3=87U * > > > --=20 *Tolga KAVUK=C3=87U* --001a1140204a426856052fe46c08 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Denis,

IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter seems like decr= eased heap usage. I will run longer tests to check heap behaviour.

Also i need another help with thread's created by = ignite. I found out that ignite creates a cleanup thread named "ttl-cl= eanup-worker" for each cache.=C2=A0 But when cache is destroyed, clean= up thread does not deleted. Instead it waits sleeping state at all.=C2=A0<= /div>

My first question is that , is it possible to decr= ease thread count with a configuration, like "thread pool with x threa= ds" for all caches.=C2=A0
Secondly, is "unremoved threa= ds" are expected behaviour.

Thanks.

On Thu, Apr 7, = 2016 at 2:40 PM, Denis Magda <dmagda@gridgain.com> wrote:<= br>
=20 =20 =20
Hi Tolga,

GridDhtPartitionTopologyImpl is created per cache. If you destroy a cache this object should be GCed. However you should use cache.destroy() for that.

Please also make sure that you make "live set" heap dumps onl= y. Try to perform GC explicitly before making the dump because a collector may clean dead objects much later depending on its heuristics.

--
Denis

On 4/7/2016 8:27 AM, Tolga Kavukcu wrote:
Hi Denis,

Thanks for the response. I will try IGNITE_ATOMIC_CACHE_DELETE_HISTO= RY_SIZE=C2=A0parameter. The screnshots taken from eclipse memory analyser which opens and analyses heap dump. I understand heap requirement for wrapping and indexing off-heap entry positions. But also found out that instances of=C2=A0org.apache.ignite.internal.processo= rs.cache.distributed.dht.GridDhtPartitionTopologyImpl is constantly increasing within jvm.


I also create and destroy so many small caches during the lifecycle, do you think that it is possible to destroyed caches leaves a footprint in heap.

The previous scrreenshots was dominator tree view of memory analyser. I attached again with headers.

=C2=A0You can see that each of GridDhtPartitionTopologyImpl us= es 20mb~ heap. And there are 72 instances of GridDhtPartitionTopologyImpl living.

Also i attached screenshots of leak suspect report of memory analyzer, which is taken periodically. You an see that instances of=C2=A0GridDhtPartitionTopologyImpl keeps increasing.=C2=A0

Any ideas or suggestions?

On Wed, Apr 6, 2016 at 6:00 PM, Denis Magda <dmagda@gridgain.com> wrote:
Hi Tolga,

GridDhtPartitionTopologyImpl contains list of partitions that belong to a specific node. In case of offheap caches each partition (concurrent map) contains set of wrappers around keys->values, stored offheap. The wrapper holds information that's needed to unswap a value or a key to Java heap from offheap when required by a user application.
So Ignite requires extra space for internal needs even when offheap mode is used.

I would recommend you trying to reduce IgniteSystemProperties.IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZ= E. This is the size of the queue that keeps deleted entries for internal needs as well.
https://apacheignite.readme.io/v1.5/docs/capaci= ty-planning

BTW, could you explain what columns from your screenshot mean exactly? What tool did you use to create the memory snapshot?

--
Denis



On 4/6/2016 3:02 PM, Tolga Kavukcu wrote:
Hi everyone,

I use partitioned ignite cache for a very dynamic data. Means that there are many updates,deletes and puts with around 5m~ row.

So to avoid gc pauses i use off-heap mode. But when i analyse heap i see that count and heap size of=C2=A0org.apache.ignite.internal.pro= cessors.cache.distributed.dht.GridDhtPartitionTopologyImpl=C2=A0is increasing constantly.=C2=A0

Please see attached screenshots taken from mat heap dump.=C2=A0

<bean class<=
/span>=3D"org.apache.ignite.conf=
iguration.CacheConfiguration" name=3D"DEFAULT&quo=
t;>
    <property name=3D"atomicityMode" value=3D"ATOMIC&q=
uot; />
    <property name=3D"cacheMode" value=3D"PARTITIONED&=
quot; />
    <property name=3D"memoryMode" value=3D"OFFHEAP_TIE=
RED" />
    <property name=3D"backups" value=3D"1" />
    <property name=3D"affinity">
        <bean class=3D"org.apache.ignite.cache.affinity.fair.FairAffinityFunctio=
n">
            <constructor-a=
rg index=3D"0" type=3D"int&q=
uot; value=3D"6"/>
        </bean>
    </property><=
span style=3D"color:rgb(128,128,128)">
    <property name=3D"writeThrough=
" value=3D"false" />
    <property name=3D"writeBehindEnabled" value=3D"fal=
se" />
</bean>
Thanks for helping out.
=C2=A0
There are totally 1.2 heap used by GridDhtPartitionTopologyImpl, almost equals to my data size. Do you think that there are problems with configuration.

Tolga KAVUK=C3=87U





--
Tolga KAVUK=C3=87U





--
Tolga KAVUK=C3=87U

--001a1140204a426856052fe46c08--