ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Du" <shawn...@neulion.com.cn>
Subject 答复: ignite client memory issue
Date Tue, 24 Jan 2017 01:09:40 GMT
Hi Denis,

 

This is the code, more things: 

#1 We store a *table* in each cache entry. 

#2 Table has several columns, column may be a big object like more than 10k bytes. Column
implement interface Binarylizable.

 

I think you can try to produce this issue by design a class like above table. Also if you
need heap dump, I can share it also.

 

Thanks

Shawn

 

private void save(Context context)
{
    BinaryObjectBuilder builder = IgniteManager.getIgnite().binary().builder(context.cacheName);
    builder.setField(IgniteConstants.COLUMN_TIMESTAMP, context.table.getTimestamp() / 1000);
    builder.setField(IgniteConstants.COLUMN_SITE, context.table.getSite());
    builder.setField(IgniteConstants.COLUMN_PRODUCT, context.table.getProduct());
    context.table.getDimColumns().forEach(c -> builder.setField(c.getName(), c));
    builder.setField(context.table.getMeasColumn().getName(), context.table.getMeasColumn());
    IgniteCache<String, BinaryObject> cache = getCache(context);
    cache.put(generateKey(context.table), builder.build());
}

private CacheConfiguration<String, BinaryObject> createCacheConfiguration(Context context)
{
    CacheConfiguration<String, BinaryObject> config = new CacheConfiguration<>();
    List<ColumnScheme> columns = new ArrayList<>();
    columns.add(new ColumnScheme(IgniteConstants.COLUMN_TIMESTAMP, Long.class.getTypeName(),
            commonConfig.indexingTimestampEnable, false));
    columns.add(new ColumnScheme(IgniteConstants.COLUMN_SITE, commonConfig.indexingSiteEnable));
    columns.add(new ColumnScheme(IgniteConstants.COLUMN_PRODUCT, commonConfig.indexingProductEnable));
    for (Column<String> column : context.table.getDimColumns())
    {
        columns.add(new ColumnScheme(column.getName(), Column.class.getTypeName(), false));
    }
    String measName = context.table.getMeasColumn().getName();
    columns.add(new ColumnScheme(measName, Column.class.getTypeName(), false));
    config.setQueryEntities(Collections
            .singleton(IgniteManager.createEntity(String.class.getTypeName(), context.cacheName,
columns)));
    config.setName(context.cacheName);
    config.setMemoryMode(commonConfig.cacheMemoryMode);
    config.setBackups(commonConfig.backups);
    config.setStartSize(10_000);
    config.setCopyOnRead(commonConfig.copyOnRead);
    return config;
}

private IgniteCache<String, BinaryObject> getCache(Context context)
{
    IgniteCache<String, BinaryObject> cache = IgniteManager.getIgnite().cache(context.cacheName);
    if (cache == null)
    {
        cache = IgniteManager.getIgnite().getOrCreateCache(createCacheConfiguration(context));
   }
    if (context.ttl > 0)
    {
        cache = cache.withExpiryPolicy(new ModifiedExpiryPolicy(new Duration(TimeUnit.SECONDS,
context.ttl)));
    }
    cache = cache.withKeepBinary();
    return cache;
}

static class Context
{
    final long ttl;
    final Table table;
    final String cacheName;

    Context(long ttl, Table table)
    {
        this.ttl = ttl;
        this.table = table;
        this.cacheName = table.getSchema().generateCacheName();
    }
}

 

 

发件人: Denis Magda [mailto:dmagda@apache.org] 
发送时间: 2017年1月24日 3:30
收件人: user@ignite.apache.org
主题: Re: ignite client memory issue

 

Hi,

 

Share piece of the code that produces the leak.

 

—

Denis

 

On Jan 22, 2017, at 5:56 PM, Shawn Du <shawn.du@neulion.com.cn <mailto:shawn.du@neulion.com.cn>
> wrote:

 

Hi,

 

My application run overnight  and crash again after I set max heap as 2G. For I saw there
were many Future Objects,

I guess it may be caused by Async API. Now I am using Sync API. It seems that the memory issue
disappeared. 

My application memory kept at 60M.

 

cache = cache.withAsync().withKeepBinary(); -->

cache = cache.withKeepBinary();

 

I think this will not happens always, but real happened on some condition and worthy further
investigate.

 

Thanks

Shawn

 

发件人: Shawn Du [mailto:shawn.du@neulion.com.cn] 
发送时间: 2017年1月22日 17:36
收件人: user@ignite.apache.org <mailto:user@ignite.apache.org> 
主题: RE: ignite client memory issue

 

Hi,

 

I am sure there are memory leaks.   See below.  

 

Class Name                                                                               
                                                           | Objects | Shallow Heap |  Retained
Heap

---------------------------------------------------------------------------------------------------------------------------------------------

java.lang.Thread                                                                         
                                                         |     128 |       15,360 | >=
879,574,832

java.lang.ThreadLocal$ThreadLocalMap                                                     
                                 |     100 |        2,400 | >= 870,478,984

java.lang.ThreadLocal$ThreadLocalMap$Entry[]                                             
                          |     100 |      271,168 | >= 870,476,576

java.lang.ThreadLocal$ThreadLocalMap$Entry                                               
                           |  23,153 |      740,896 | >= 868,443,568

org.apache.ignite.internal.processors.cache.IgniteCacheFutureImpl                        
         |  22,444 |      359,104 | >= 867,056,800

org.apache.ignite.internal.util.future.GridFutureAdapter$ChainFuture                     
         |  22,444 |    1,436,416 | >= 866,697,704

byte[]                                                                                   
                                                                     |  26,362 |  864,837,368
| >= 864,837,368

org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture|
 22,441 |    3,411,032 | >= 861,188,352

org.apache.ignite.internal.binary.BinaryObjectImpl                                       
                             |  22,441 |      897,640 | >= 855,099,616

---------------------------------------------------------------------------------------------------------------------------------------------

 

Now my application is still running, and the memory is growing up. Please help.

 

More information:

 

Ignite version: 1.8.0

java version "1.8.0_77"

Java(TM) SE Runtime Environment (build 1.8.0_77-b03)

Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

Platform:

Linux dev-s2 4.4.8-20.46.amzn1.x86_64 #1 SMP Wed Apr 27 19:28:52 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux

 

 

This is a new issue in my application. Recently we change our code. Add below Class which
implement 

public class Column<T> implements Binarylizable

{
}

 

Thanks. Please help.

 

Shawn

 

 

发件人: Shawn Du [ <mailto:shawn.du@neulion.com.cn> mailto:shawn.du@neulion.com.cn]

发送时间: 2017年1月22日 13:52
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: Re: ignite client memory issue

 

Hi,

I assume this document for ignite server.  In my case, I set ignite work at client mode. 

Now I increase the Max Heap Size and my application is running, I will monitor the memory
usage. 

For my view, at client mode, ignite should not use too much memory.

 

Thanks

Shawn

 

发件人: Denis Magda [ <mailto:dmagda@apache.org> mailto:dmagda@apache.org] 
发送时间: 2017年1月22日 13:38
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: Re: ignite client memory issue

 

Also keep in mind that every Ignite node requires at least ~ 300 MB for its internal purposes.
This capacity planning page might be useful as well for you:

 <https://apacheignite.readme.io/docs/capacity-planning-bak> https://apacheignite.readme.io/docs/capacity-planning-bak

 

—

Denis

 

On Jan 21, 2017, at 9:29 PM, Shawn Du < <mailto:shawn.du@neulion.com.cn> shawn.du@neulion.com.cn>
wrote:

 

My heap max size is 768M. more than 500M  are consumed  by ignite.

 

Your advice is quite reasonable. I will refactor my code. 

 

Thanks

发件人: Denis Magda [ <mailto:dmagda@apache.org> mailto:dmagda@apache.org] 
发送时间: 2017年1月22日 13:18
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: Re: ignite client memory issue

 

Hi Shawn,

 

What is the maximum size of the heap? 

 

I don’t think the cache configurations can be a reason of the OOM. As a side note, there
is no reason to keep the configurations at all. Once a cache is started with a configuration
you can either keep a single reference to it and reuse by multiple app threads or get a new
one by passing the cache name into a respective Ignite method.

 

—

Denis

 

On Jan 21, 2017, at 9:03 PM, Shawn Du < <mailto:shawn.du@neulion.com.cn> shawn.du@neulion.com.cn>
wrote:

 

I review the code, find a never released HashMap which store some cache configurations.

private Map<TableSchema, CacheConfiguration<String, BinaryObject>> cacheConfigurations
= new HashMap<>();

I cache these configurations for performance consideration. 

I get ignite cache by calling:

IgniteCache<String, BinaryObject> cache = IgniteManager.getIgnite().getOrCreateCache(configuration);

 

My question:

If cache configuration doesn’t release, it will prevent some memeory/cache entries/something
else release?

 

Thanks

Shawn

 

发件人: Shawn Du [ <mailto:shawn.du@neulion.com.cn> mailto:shawn.du@neulion.com.cn]

发送时间: 2017年1月22日 9:09
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: ignite client memory issue

 

Hi,

 

My ignite client died many times recently because of OOM. Of course, I can increase the Max
heap size. But I want to know why these memory are not released.

This is part of analysis of Eclipse Memory Analyzer. My ignite client use Async cache API.
please help.

 

Class Name                                                                               
         | Objects | Shallow Heap |  Retained Heap

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

java.lang.Thread                                                                         
           |     109 |       13,080 | >= 561,086,296

java.lang.ThreadLocal$ThreadLocalMap                                                     
            |      98 |        2,352 | >= 537,825,592

java.lang.ThreadLocal$ThreadLocalMap$Entry[]                                             
             |      98 |      139,808 | >= 537,823,232

java.lang.ThreadLocal$ThreadLocalMap$Entry                                               
             |  15,202 |      486,464 | >= 535,750,616

org.apache.ignite.internal.processors.cache.IgniteCacheFutureImpl                        
                  |  14,476 |      231,616 | >= 534,866,208

org.apache.ignite.internal.util.future.GridFutureAdapter$ChainFuture                     
                    |  14,476 |      926,464 | >= 534,634,600

org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture
             |  14,466 |    2,198,832 | >= 531,444,584

byte[]                                                                                   
          |  18,438 |  530,369,360 | >= 530,369,360

org.apache.ignite.internal.binary.BinaryObjectImpl                                       
                 |  14,466 |      578,640 | >= 527,749,368

 

Thanks

Shawn

 


Mime
View raw message