ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isaeed Mohanna <isae...@gmail.com>
Subject Re: Cache Memory Behavior \ GridDhtLocalPartition
Date Sun, 13 Nov 2016 13:14:22 GMT
Hi
There is no eviction policy since entries in the caches are removed by my
application. (calling IgniteCache.remove ).

Digging through the core dump I can see that most resident items are cache
entries were the cachecontext.cachename points to EventsCache, as i have
mentioned before this cache has very frequent writes and deletions of
events (i am using remove to delete the events), however this cache is also
atomic,partitioned and have a backup of at least one so in case a node
fails the event is not lost. when calling remove on a cache, is the backup
of an entry removed as well? is it possible that the backup is not being
cleaned?

Currently i am using the default garbage collector settings, i can't see
any spikes in performance due to GC, since i experience memory outage in
several days i am not sure i am collecting data more than the GC is able to
claim, I will try manually performing a GC when the system is about to
crash to see wither forcing a GC will clean the memory.

Thank you for ur help

On Fri, Nov 11, 2016 at 11:59 AM, Andrey Mashenkov <amashenkov@gridgain.com>
wrote:

> Hi Isaeed Mohanna,
>
> I don't see any eviction or expired policy configured. Is entry deletion
> performed by you application?
>
> Have you try to detect which of caches id grows unexpectedly?
> Have you analyse GC logs or tried to tune GC? Actually, you can putting
> data faster as garbage is collecting. This page may be helpful
> http://apacheignite.gridgain.org/v1.7/docs/performance-tips#tune-garbage-
> collection.
>
> Also you can get profile (with e.g. JavaFlightRecorder) of grid under load
> to understand what is really going on.
>
> Please let me know, if there are any issues.
>
>
>
> On Thu, Nov 10, 2016 at 10:10 AM, Isaeed Mohanna <isaeedm@gmail.com>
> wrote:
>
>> Hi
>> My cache configurations appear below.
>>
>> // Cache 1 - a cache of ~15 entities that has a date stamp that is
>> updated every 30 - 120 seconds
>> CacheConfiguration<?, ?> Cache1Cfg = new CacheConfiguration<>();
>> Cache1Cfg cheCfg.setName("Cache1Name");
>> Cache1Cfg .setCacheMode(CacheMode.REPLICATED);
>> Cache1Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> Cache1Cfg .setStartSize(50);
>>
>> // Cache 2 - A cache used as an ignite queue with frequent inserts and
>> removal from the queue
>> CacheConfiguration<?, ?> Cache2Cfg = new CacheConfiguration<>();
>> Cache2Cfg .setName("Cache2Name");
>> Cache2Cfg .setCacheMode(CacheMode.REPLICATED);
>> Cache2Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>
>> // Cache 3 - hundreds of entities updated daily
>> CacheConfiguration<?, ?> Cache3Cfg = new CacheConfiguration<>();
>> Cache3Cfg .setName("Cache3Name");
>> Cache3Cfg .setCacheMode(CacheMode.REPLICATED);
>> Cache3Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> Cache3Cfg .setIndexedTypes(UUID.class, SomeClass.class);
>>
>> // Cache 4 - Cache with very few writes and reads
>> CacheConfiguration<?, ?> Cache4Cfg = new CacheConfiguration<>();
>> Cache4Cfg .setName("Cache4Name");
>> Cache4Cfg .setCacheMode(CacheMode.REPLICATED);
>> Cache4Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>
>> // Events Cache - cache with very frequent writes and delete, acts as
>> events queue
>> CacheConfiguration<?, ?> eventsCacheConfig= new CacheConfiguration<>();
>> eventsCacheConfig.setName("EventsCache");
>> eventsCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> eventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> eventsCacheConfig.setIndexedTypes(UUID.class, SomeClass.class);
>> eventsCacheConfig.setBackups(1);
>> eventsCacheConfig.setOffHeapMaxMemory(0);
>>
>> // Failed Events Cache - cache with less writes and reads stores failed
>> events
>> CacheConfiguration<?, ?> failedEventsCacheConfig = new
>> CacheConfiguration<>();
>> failedEventsCacheConfig.setName("FailedEventsCache");
>> failedEventsCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> failedEventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> failedEventsCacheConfig.setIndexedTypes(UUID.class, EventEntity.class);
>> failedEventsCacheConfig.setBackups(1);
>> failedEventsCacheConfig.setOffHeapMaxMemory(0);
>>
>> // In addition i have one atomic reference
>> AtomicConfiguration atomicCfg = new AtomicConfiguration();
>> atomicCfg.setCacheMode(CacheMode.REPLICATED);
>> Thanks again
>>
>> On Wed, Nov 9, 2016 at 5:26 PM, Andrey Mashenkov <amashenkov@gridgain.com
>> > wrote:
>>
>>> Hi Isaeed Mohanna,
>>>
>>> Would you please provide your cache configurations?
>>>
>>>
>>> On Wed, Nov 9, 2016 at 5:37 PM, Isaeed Mohanna <isaeedm@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>> i have an ignite 1.7.0 cluster with 3 nodes running , i have 3
>>>> PARTITIONED
>>>> ATOMIC CACHES and 2 REPLICATED ATOMIC CACHES, Most of these caches are
>>>> populated with events data, so each cache entry is short lived its
>>>> inserted,
>>>> processed later by some task and removed. so the caches are pretty much
>>>> very
>>>> dynamic.
>>>> Recently the load in our system has increased (more events were
>>>> received and
>>>> generated) and we started experiencing out of memory fails once in while
>>>> (several days depending on machine size).
>>>> I have created several heap dumps and noticed the largest retained
>>>> objects
>>>> in memory is by the following classes: GridDhtLocalPartition,
>>>> ConccurentHashMap8,ConccurentHashMap8$Node[].
>>>> I can see the GridDhtLocalPartition has a ConccurentHashMap8 so most
>>>> likely
>>>> all three reference the same thing.
>>>> My question what is this class and why does it retain memory, entities
>>>> in my
>>>> caches are usually short lived (several minutes in most caches) so i
>>>> would
>>>> expect the memory to be released? any hints on how to continue my
>>>> investigation would be great.
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-ignite-users.705
>>>> 18.x6.nabble.com/Cache-Memory-Behavior-GridDhtLocalPartition
>>>> -tp8835.html
>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>

Mime
View raw message