hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-16193) Memory leak when putting plenty of duplicated cells
Date Thu, 07 Jul 2016 16:48:11 GMT
Yu Li created HBASE-16193:

             Summary: Memory leak when putting plenty of duplicated cells
                 Key: HBASE-16193
                 URL: https://issues.apache.org/jira/browse/HBASE-16193
             Project: HBase
          Issue Type: Bug
            Reporter: Yu Li
            Assignee: Yu Li

Recently we suffered from a weird problem that RS heap size could not reduce much even after
FullGC, and it kept FullGC and could hardly serve any request. After debugging for days, we
found the root cause: we won't count in the allocated memory in MSLAB chunk when adding duplicated
cells (including put and delete). We have below codes in {{AbstractMemStore#add}} (or {{DefaultMemStore#add}}
for branch-1):
  public long add(Cell cell) {
    Cell toAdd = maybeCloneWithAllocator(cell);
    return internalAdd(toAdd);
where we will allocate memory in MSLAB (if using) chunk for the cell first, and then call
{{internalAdd}}, where we could see below codes in {{Segment#internalAdd}} (or {{DefaultMemStore#internalAdd}}
for branch-1):
  protected long internalAdd(Cell cell) {
    boolean succ = getCellSet().add(cell);
    long s = AbstractMemStore.heapSizeChange(cell, succ);
    updateMetaInfo(cell, s);
    return s;
So if we are writing a duplicated cell, we assume there's no heap size change, while actually
the chunk size is taken (referenced).

Let's assume this scenario, that there're huge amount of writing on the same cell (same key,
different values), which is not that special in MachineLearning use case, and there're also
few normal writes, and after some long time, it's possible that we have many chunks with kvs
like: {{cellA, cellB, cellA, cellA, .... cellA}}, that we only counts 2 cells for each chunk,
but actually the chunk is full. So the devil comes, that we think it's still not hitting flush
size, while there's already GBs heapsize taken.

There's also a more extreme case, that we only writes a single cell over and over again and
fills one chunk quickly. Ideally the chunk should be cleared by GC, but unfortunately we have
kept a redundant reference in {{HeapMemStore#chunkQueue}}, which is useless when we're not
using chunkPool by default.

This is the umbrella to describe the problem, and I'll open two sub-JIRAs to resolve the above
two issues separately.

This message was sent by Atlassian JIRA

View raw message