ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <amashen...@gridgain.com>
Subject Re: Cache Memory Behavior \ GridDhtLocalPartition
Date Mon, 14 Nov 2016 14:50:51 GMT
Hi,

On remove operation entry should be removed from primary node and backup
nodes as well.
Can you reproduce the issue? Can you check if entry was removed only from
primary node and exists on backup, e.g. using constant affinity function?

I think its possible that the backup is not being cleaned, due to key
serialization issues. Would you provide key class implementation?

On Sun, Nov 13, 2016 at 4:14 PM, Isaeed Mohanna <isaeedm@gmail.com> wrote:

> Hi
> There is no eviction policy since entries in the caches are removed by my
> application. (calling IgniteCache.remove ).
>
> Digging through the core dump I can see that most resident items are cache
> entries were the cachecontext.cachename points to EventsCache, as i have
> mentioned before this cache has very frequent writes and deletions of
> events (i am using remove to delete the events), however this cache is also
> atomic,partitioned and have a backup of at least one so in case a node
> fails the event is not lost. when calling remove on a cache, is the backup
> of an entry removed as well? is it possible that the backup is not being
> cleaned?
>
> Currently i am using the default garbage collector settings, i can't see
> any spikes in performance due to GC, since i experience memory outage in
> several days i am not sure i am collecting data more than the GC is able to
> claim, I will try manually performing a GC when the system is about to
> crash to see wither forcing a GC will clean the memory.
>
> Thank you for ur help
>
> On Fri, Nov 11, 2016 at 11:59 AM, Andrey Mashenkov <
> amashenkov@gridgain.com> wrote:
>
>> Hi Isaeed Mohanna,
>>
>> I don't see any eviction or expired policy configured. Is entry deletion
>> performed by you application?
>>
>> Have you try to detect which of caches id grows unexpectedly?
>> Have you analyse GC logs or tried to tune GC? Actually, you can putting
>> data faster as garbage is collecting. This page may be helpful
>> http://apacheignite.gridgain.org/v1.7/docs/performan
>> ce-tips#tune-garbage-collection.
>>
>> Also you can get profile (with e.g. JavaFlightRecorder) of grid under
>> load to understand what is really going on.
>>
>> Please let me know, if there are any issues.
>>
>>
>>
>> On Thu, Nov 10, 2016 at 10:10 AM, Isaeed Mohanna <isaeedm@gmail.com>
>> wrote:
>>
>>> Hi
>>> My cache configurations appear below.
>>>
>>> // Cache 1 - a cache of ~15 entities that has a date stamp that is
>>> updated every 30 - 120 seconds
>>> CacheConfiguration<?, ?> Cache1Cfg = new CacheConfiguration<>();
>>> Cache1Cfg cheCfg.setName("Cache1Name");
>>> Cache1Cfg .setCacheMode(CacheMode.REPLICATED);
>>> Cache1Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>> Cache1Cfg .setStartSize(50);
>>>
>>> // Cache 2 - A cache used as an ignite queue with frequent inserts and
>>> removal from the queue
>>> CacheConfiguration<?, ?> Cache2Cfg = new CacheConfiguration<>();
>>> Cache2Cfg .setName("Cache2Name");
>>> Cache2Cfg .setCacheMode(CacheMode.REPLICATED);
>>> Cache2Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>>
>>> // Cache 3 - hundreds of entities updated daily
>>> CacheConfiguration<?, ?> Cache3Cfg = new CacheConfiguration<>();
>>> Cache3Cfg .setName("Cache3Name");
>>> Cache3Cfg .setCacheMode(CacheMode.REPLICATED);
>>> Cache3Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>> Cache3Cfg .setIndexedTypes(UUID.class, SomeClass.class);
>>>
>>> // Cache 4 - Cache with very few writes and reads
>>> CacheConfiguration<?, ?> Cache4Cfg = new CacheConfiguration<>();
>>> Cache4Cfg .setName("Cache4Name");
>>> Cache4Cfg .setCacheMode(CacheMode.REPLICATED);
>>> Cache4Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>>
>>> // Events Cache - cache with very frequent writes and delete, acts as
>>> events queue
>>> CacheConfiguration<?, ?> eventsCacheConfig= new CacheConfiguration<>();
>>> eventsCacheConfig.setName("EventsCache");
>>> eventsCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>>> eventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>> eventsCacheConfig.setIndexedTypes(UUID.class, SomeClass.class);
>>> eventsCacheConfig.setBackups(1);
>>> eventsCacheConfig.setOffHeapMaxMemory(0);
>>>
>>> // Failed Events Cache - cache with less writes and reads stores failed
>>> events
>>> CacheConfiguration<?, ?> failedEventsCacheConfig = new
>>> CacheConfiguration<>();
>>> failedEventsCacheConfig.setName("FailedEventsCache");
>>> failedEventsCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>>> failedEventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>> failedEventsCacheConfig.setIndexedTypes(UUID.class, EventEntity.class);
>>> failedEventsCacheConfig.setBackups(1);
>>> failedEventsCacheConfig.setOffHeapMaxMemory(0);
>>>
>>> // In addition i have one atomic reference
>>> AtomicConfiguration atomicCfg = new AtomicConfiguration();
>>> atomicCfg.setCacheMode(CacheMode.REPLICATED);
>>> Thanks again
>>>
>>> On Wed, Nov 9, 2016 at 5:26 PM, Andrey Mashenkov <
>>> amashenkov@gridgain.com> wrote:
>>>
>>>> Hi Isaeed Mohanna,
>>>>
>>>> Would you please provide your cache configurations?
>>>>
>>>>
>>>> On Wed, Nov 9, 2016 at 5:37 PM, Isaeed Mohanna <isaeedm@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi
>>>>> i have an ignite 1.7.0 cluster with 3 nodes running , i have 3
>>>>> PARTITIONED
>>>>> ATOMIC CACHES and 2 REPLICATED ATOMIC CACHES, Most of these caches are
>>>>> populated with events data, so each cache entry is short lived its
>>>>> inserted,
>>>>> processed later by some task and removed. so the caches are pretty
>>>>> much very
>>>>> dynamic.
>>>>> Recently the load in our system has increased (more events were
>>>>> received and
>>>>> generated) and we started experiencing out of memory fails once in
>>>>> while
>>>>> (several days depending on machine size).
>>>>> I have created several heap dumps and noticed the largest retained
>>>>> objects
>>>>> in memory is by the following classes: GridDhtLocalPartition,
>>>>> ConccurentHashMap8,ConccurentHashMap8$Node[].
>>>>> I can see the GridDhtLocalPartition has a ConccurentHashMap8 so most
>>>>> likely
>>>>> all three reference the same thing.
>>>>> My question what is this class and why does it retain memory, entities
>>>>> in my
>>>>> caches are usually short lived (several minutes in most caches) so i
>>>>> would
>>>>> expect the memory to be released? any hints on how to continue my
>>>>> investigation would be great.
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://apache-ignite-users.705
>>>>> 18.x6.nabble.com/Cache-Memory-Behavior-GridDhtLocalPartition
>>>>> -tp8835.html
>>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message