ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tolga Kavukcu <kavukcu.to...@gmail.com>
Subject Re: Unexpected heap usage by GridDhtPartitionTopologyImpl
Date Fri, 08 Apr 2016 08:11:24 GMT
Hi Denis,

Yes we don't need to have expiration policy, so setting
CacheConfiguraiton.setEagerTtl to false solved this problem.

So we are testing the whole system, we also found out that using a cache
with writeBehind enabled causes a new thread creation for each one. So if
we think about future plans and possibilities, it's not a best practice to
have increasing number of threads within jvm.

We wonder that if there is a option to use a thread pool for writeBehind
jobs. If there is not, we could implement it for the community. So if you
can guide us where to start, i would be glad :)

Thanks.

On Fri, Apr 8, 2016 at 1:54 AM, Denis Magda <dmagda@gridgain.com> wrote:

> Tolga,
>
> The cleanup threads "ttl-cleanup-workers" are used to eagerly remove
> expired cache entries. Expiration policy can be set either a cache wide in
> CacheConfiguration or can be used later with cache.withExpirePolicy(...)
> calls.
> I failed to reproduce your case. What I've done is started 30 caches and
> destroyed all of them later. Visual VM showed that all
> "ttl-cleanup-workers" were stopped successfully.
> What Ignite version do you use?
>
> In any case if you are not planing to use expiration policy you can set
> CacheConfiguraiton.setEagerTtl to false and the ttl workers Threads won't
> be created at all.
>
> Regards,
> Denis
>
>
> On 4/7/2016 3:43 PM, Tolga Kavukcu wrote:
>
> Hi Denis,
>
> IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter seems like decreased
> heap usage. I will run longer tests to check heap behaviour.
>
> Also i need another help with thread's created by ignite. I found out that
> ignite creates a cleanup thread named "ttl-cleanup-worker" for each cache.
> But when cache is destroyed, clean up thread does not deleted. Instead it
> waits sleeping state at all.
>
> My first question is that , is it possible to decrease thread count with a
> configuration, like "thread pool with x threads" for all caches.
> Secondly, is "unremoved threads" are expected behaviour.
>
> Thanks.
>
> On Thu, Apr 7, 2016 at 2:40 PM, Denis Magda <dmagda@gridgain.com> wrote:
>
>> Hi Tolga,
>>
>> GridDhtPartitionTopologyImpl is created per cache. If you destroy a cache
>> this object should be GCed. However you should use cache.destroy() for that.
>>
>> Please also make sure that you make "live set" heap dumps only. Try to
>> perform GC explicitly before making the dump because a collector may clean
>> dead objects much later depending on its heuristics.
>>
>> --
>> Denis
>>
>> On 4/7/2016 8:27 AM, Tolga Kavukcu wrote:
>>
>> Hi Denis,
>>
>> Thanks for the response. I will try IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter.
>> The screnshots taken from eclipse memory analyser which opens and analyses
>> heap dump. I understand heap requirement for wrapping and indexing off-heap
>> entry positions. But also found out that instances of *org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl
>> *is constantly increasing within jvm.
>>
>>
>> I also create and destroy so many small caches during the lifecycle, do
>> you think that it is possible to destroyed caches leaves a footprint in
>> heap.
>>
>> The previous scrreenshots was dominator tree view of memory analyser. I
>> attached again with headers.
>>
>>  You can see that each of GridDhtPartitionTopologyImpl uses 20mb~ heap.
>> And there are 72 instances of GridDhtPartitionTopologyImpl living.
>>
>> Also i attached screenshots of leak suspect report of memory analyzer,
>> which is taken periodically. You an see that instances of *GridDhtPartitionTopologyImpl
>> keeps increasing. *
>>
>> Any ideas or suggestions?
>>
>> On Wed, Apr 6, 2016 at 6:00 PM, Denis Magda < <dmagda@gridgain.com>
>> dmagda@gridgain.com> wrote:
>>
>>> Hi Tolga,
>>>
>>> GridDhtPartitionTopologyImpl contains list of partitions that belong to
>>> a specific node. In case of offheap caches each partition (concurrent map)
>>> contains set of wrappers around keys->values, stored offheap. The wrapper
>>> holds information that's needed to unswap a value or a key to Java heap
>>> from offheap when required by a user application.
>>> So Ignite requires extra space for internal needs even when offheap mode
>>> is used.
>>>
>>> I would recommend you trying to reduce
>>> IgniteSystemProperties.IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. This is the
>>> size of the queue that keeps deleted entries for internal needs as well.
>>> https://apacheignite.readme.io/v1.5/docs/capacity-planning
>>>
>>> BTW, could you explain what columns from your screenshot mean exactly?
>>> What tool did you use to create the memory snapshot?
>>>
>>> --
>>> Denis
>>>
>>>
>>>
>>> On 4/6/2016 3:02 PM, Tolga Kavukcu wrote:
>>>
>>> Hi everyone,
>>>
>>> I use partitioned ignite cache for a very dynamic data. Means that there
>>> are many updates,deletes and puts with around 5m~ row.
>>>
>>> So to avoid gc pauses i use off-heap mode. But when i analyse heap i see
>>> that count and heap size of
>>> *org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl*
is
>>> increasing constantly.
>>>
>>> Please see attached screenshots taken from mat heap dump.
>>>
>>> <bean class="org.apache.ignite.configuration.CacheConfiguration" name="DEFAULT">
   <property name="atomicityMode" value="ATOMIC" />    <property name="cacheMode"
value="PARTITIONED" />    <property name="memoryMode" value="OFFHEAP_TIERED" /> 
  <property name="backups" value="1" />    <property name="affinity">        <bean
class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction">            <constructor-arg
index="0" type="int" value="6"/>        </bean>    </property>    <property
name="writeThrough" value="false" />    <property name="writeBehindEnabled" value="false"
/></bean>
>>>
>>> Thanks for helping out.
>>>
>>> There are totally 1.2 heap used by GridDhtPartitionTopologyImpl, almost
>>> equals to my data size. Do you think that there are problems with
>>> configuration.
>>>
>>>
>>> *Tolga KAVUKÇU *
>>>
>>>
>>>
>>
>>
>> --
>>
>> *Tolga KAVUKÇU *
>>
>>
>>
>
>
> --
>
> *Tolga KAVUKÇU *
>
>
>


-- 

*Tolga KAVUKÇU*

Mime
View raw message