ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Du" <shawn...@neulion.com.cn>
Subject 答复: 答复: ignite used too much memory
Date Thu, 27 Oct 2016 07:33:01 GMT
BTW, I use ONHEAD_TIERED memory mode. 

 

I don’t enable off heap.

 

Thanks

Shawn

 

发件人: Shawn Du [mailto:shawn.du@neulion.com.cn] 
发送时间: 2016年10月27日 15:30
收件人: 'user@ignite.apache.org'
主题: 答复: 答复: ignite used too much memory

 

Hi Andrey Mashenkov,

 

I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go better but not resolved.

 

This is my cache state,  I think it is full, for code like this:

 

config.setSwapEnabled(swapEnable);
if (swapEnable)
{
    EvictionPolicy policy = new FifoEvictionPolicy(10000);
    config.setEvictionPolicy(policy);
}

 

new caches should be in memory and the old will be evicted into disk. The memory should stopping
growing.

After my test, it is not the case. It is still keep growing fast, at speed of 500M/30minutes.


I can understand that there will need some extra memory to maintain something, but it leaks
too fast.

 

Thanks

Shawn

 

 

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| onlinesessioncount_by_cityid(@c1)   | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| onlineviewcount_by_cityid(@c3)      | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| onlineviewcount_by_regionid(@c4)    | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| onlinevisitorcount_by_cityid(@c5)   | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| videoviewcount_by_cityid(@c7)       | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+

| videoviewcount_by_regionid(@c8)     | PARTITIONED | 2     | min: 0 (0 / 0)             
  | min: 0    | min: 0    | min: 0    | min: 0    |

|                                     |             |       | avg: 5000.00 (5000.00 / 0.00)
| avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |

|                                     |             |       | max: 10000 (10000 / 0)     
  | max: 0    | max: 0    | max: 0    | max: 0    |

+-------------------------------------------------------------------------------------------------------------------------------------------+

 

发件人: Shawn Du [mailto:shawn.du@neulion.com.cn] 
发送时间: 2016年10月27日 9:17
收件人: user@ignite.apache.org <mailto:user@ignite.apache.org> 
主题: 答复: 答复: ignite used too much memory

 

Hi Andrey Mashenkov,

 

Thanks, I will have a try and apply these locally.

 

Do you know 1.8’s release plan? 

 

Thanks

Shawn 

 

发件人: Andrey Mashenkov [ <mailto:amashenkov@gridgain.com> mailto:amashenkov@gridgain.com]

发送时间: 2016年10月26日 20:53
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: Re: 答复: ignite used too much memory

 

Hi, Shawn Du

 

It seems you faced next 2 bugs.

 

First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has a fix, but it
is not merged to master yet. See:  <https://issues.apache.org/jira/browse/IGNITE-3840>
https://issues.apache.org/jira/browse/IGNITE-3840.

Second bug: TTL Manager continue track evicted (and removed) entries,  <https://issues.apache.org/jira/browse/IGNITE-3948>
https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for merge, but still
is not present in master.

 

You can try to merge them locally or wait until they will be available in master.

 

On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <shawn.du@neulion.com.cn <mailto:shawn.du@neulion.com.cn>
> wrote:

Hi experts,

 

Can anyone help to explain ignite memory model?

 

Now I tried following ways, but no effect.

1)       Remove all indexes.

2)       enable swap. I see more than 800M’s data are stored in swapspace directory.

3)       Don’t cache short life entries. All cache with fifo evict policy and max size is
10k.

 

After running for a while, I have a check the caches, there were only 20k entries, but the
memory still grow up?

 

Please help.

 

Thanks

Shawn  

 

 

发件人: Shawn Du [mailto: <mailto:shawn.du@neulion.com.cn> shawn.du@neulion.com.cn]

发送时间: 2016年10月26日 17:17
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: 答复: ignite used too much memory

 

Hi 

 

This is the output of jmap histo:live <pid>.

 

Any useful information?  There are about 300k cache entries. Also I enable swap. 

 

Each entry costs 10K  memory in average. But each entry only has 50 bytes at most.

 

Any help will be appreciated.

 

num     #instances         #bytes  class name

----------------------------------------------

   1:      21642240      519413760  org.apache.ignite.internal.util.GridCircularBuffer$Item

   2:         49027      405647408  [Lorg.jsr166.ConcurrentHashMap8$Node;

   3:       6056069      304233144  [B

   4:       5754322      279563064  [C

   5:      11376526      273036624  java.util.concurrent.ConcurrentSkipListMap$Node

   6:       5688514      227540560  org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap

   7:       5684293      181897376  org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper

   8:       5333102      170659264  java.util.concurrent.ConcurrentHashMap$Node

   9:       5325117      170403744  org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue

  10:       5754728      138113472  java.lang.String

  11:       5689577      136549848  java.util.concurrent.ConcurrentSkipListMap$Index

  12:       5686353      136472472  org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl

  13:       5325117      127802808  org.apache.ignite.spi.swapspace.SwapKey

  14:       5690452       91047232  org.h2.value.ValueString

  15:         85112       87930752  [Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;

  16:       1515799       48505568  java.util.HashMap$Node

  17:        268818       42628512  [Ljava.lang.Object;

  18:         13467       35031504  [Ljava.util.concurrent.ConcurrentHashMap$Node;

  19:        361155       23113920  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry

  20:       1226922       19630752  java.lang.Integer

  21:        102183       15207336  [Ljava.util.HashMap$Node;

  22:        363397       14535880  org.apache.ignite.internal.binary.BinaryObjectImpl

  23:        395583       12658656  org.jsr166.ConcurrentHashMap8$Node

  24:        359176       11493632  org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras

  25:        363397        8721528  org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject

  26:        257615        6182760  java.util.concurrent.atomic.AtomicLong

  27:        170775        5464800  java.util.concurrent.locks.ReentrantLock$NonfairSync

  28:         85112        5447168  org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition

  29:         85112        5447168  org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1

  30:        103188        4953024  java.util.HashMap

  31:         87751        3510040  org.jsr166.ConcurrentHashMap8

  32:        136382        3273168  org.jsr166.ConcurrentLinkedDeque8$Node

  33:         90699        2902368  org.jsr166.LongAdder8

  34:         87277        2792864  java.lang.ref.WeakReference

  35:        170663        2730608  java.util.concurrent.locks.ReentrantLock

  36:         85112        2723584  org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl

  37:         85112        2723584  org.apache.ignite.internal.util.GridCircularBuffer

  38:        134255        2148080  org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl

  39:         85304        2047296  java.util.concurrent.CopyOnWriteArrayList

  40:         10900        1693000  [Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;

  41:         96594        1545504  java.util.HashSet

  42:         86086        1377376  java.util.concurrent.atomic.AtomicInteger

  43:         85435        1366960  java.util.HashMap$KeySet

  44:         85002        1360032  org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue

  45:         44295        1063080  java.util.ArrayList

  46:         13513         864832  java.util.concurrent.ConcurrentHashMap

  47:         21090         843600  org.apache.ignite.internal.processors.cache.version.GridCacheVersion

  48:           160         653824  [Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition;

  49:          5664         637120  java.lang.Class

  50:          7047         620136  java.lang.reflect.Method

  51:         10900         610400  org.jsr166.ConcurrentLinkedHashMap$Segment

  52:         12528         601344  java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync

  53:         11566         555168  org.h2.expression.ExpressionColumn

  54:         16698         400752  java.lang.Long

  55:         12522         400704  javax.management.MBeanAttributeInfo

  56:          7878         381504  [I

  57:          1767         310992  org.h2.command.dml.Select

  58:          6162         295776  org.apache.ignite.internal.processors.affinity.GridAffinityAssignment

  59:          7223         288920  java.util.LinkedHashMap$Entry

  60:          9276         222624  org.h2.expression.ValueExpression

  61:          6912         221184  org.h2.expression.Comparison

  62:          3042         219024  java.lang.reflect.Field

  63:         12529         200464  java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock

  64:         12529         200464  java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter

  65:         12529         200464  java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock

  66:          1767         197904  org.h2.table.TableFilter

  67:          2116         186208  org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot

  68:          1767         169632  org.h2.jdbc.JdbcPreparedStatement

  69:          5145         164640  org.h2.expression.ConditionAndOr

  70:          7674         162472  [Ljava.lang.Class;

  71:          4654         148928  org.h2.expression.Alias

  72:          1726         137328  [S

  73:          1767         127224  org.h2.index.IndexCursor

  74:          7800         124800  java.lang.Object

  75:          1360         119680  org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock

  76:          1767         113088  org.h2.jdbc.JdbcResultSet

  77:          1992         111552  java.util.LinkedHashMap

  78:          4228         101472  org.h2.value.ValueLong

  79:          2804          89728  java.lang.ThreadLocal$ThreadLocalMap$Entry

  80:           309          88992  org.apache.ignite.configuration.CacheConfiguration

  81:          1767          84816  org.h2.command.CommandContainer

  82:          1172          84384  org.h2.expression.Aggregate

  83:           777          80808  org.h2.table.Column

  84:          2502          80064  java.util.concurrent.ConcurrentSkipListMap$HeadIndex

  85:          2939          79912  [Lorg.h2.expression.Expression;

  86:          1870          73152  [Lorg.h2.table.IndexColumn;

  87:          1798          71920  java.lang.ref.SoftReference

  88:            17          69904  [Ljava.nio.ByteBuffer;

  89:          2166          69312  org.apache.ignite.internal.GridLoggerProxy

  90:          1715          68600  org.h2.expression.Operation

  91:            64          66560  [Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot;

  92:            20          65920  [Ljava.nio.channels.SelectionKey;

 

 

 

 

发件人: Shawn Du [ <mailto:shawn.du@neulion.com.cn> mailto:shawn.du@neulion.com.cn]

发送时间: 2016年10月26日 13:25
收件人:  <mailto:user@ignite.apache.org> user@ignite.apache.org
主题: ignite used too much memory

 

Hi,

 

In my ignite server, I have several caches, each cache has about 10k entries. 

I build the entry using binary object. Each entry just has 3 or 4 fields, each field is short,
less than 20 bytes. But I enable index for each field.

Most entry has set expired time. The expired time is short, about 90 seconds. 

After run for 2 hours, 8G memory are used, and the ignite run out of memory. 

I build the ignite from source code, and use yesterdays’ github master branch code.

 

My questions:

1)       Does the expired data release the memory?

2)       How ignite build the index? How much memory it cost?

 

Thanks

Shawn

 

 

 

 


Mime
View raw message