hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11425) Cell/DBB end-to-end on the read-path
Date Fri, 16 Oct 2015 20:58:08 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HBASE-11425:
    Attachment: ram.log

The log is not that interesting. It does not even have the eviction issue.  Just shows us
struggling w/ GC.  Finally we OOME here:

2015-10-16T13:16:42.032-0700: [Full GC (Allocation Failure) 2015-10-16T13:16:42.032-0700:
[CMS: 15276447K->15276422K(15276480K), 4.9700400 secs] 16273215K->16273119K(16273280K),
[Metaspace: 48934K->48934K(1093632K)], 4.9774537 secs] [Times: user=4.97 sys=0.00, real=4.98
2015-10-16T13:16:47.012-0700: [Full GC (Allocation Failure) 2015-10-16T13:16:47.012-0700:
[CMS: 15276422K->15276422K(15276480K), 1.2150090 secs] 16273204K->16273186K(16273280K),
[Metaspace: 48901K->48901K(1093632K)], 1.2151393 secs] [Times: user=1.21 sys=0.00, real=1.22
2015-10-16T13:16:48.227-0700: [Full GC (Allocation Failure) 2015-10-16T13:16:48.227-0700:
[CMS: 15276422K->15276422K(15276480K), 1.2185531 secs] 16273186K->16273186K(16273280K),
[Metaspace: 48901K->48901K(1093632K)], 1.2186671 secs] [Times: user=1.22 sys=0.00, real=1.21
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 16941"...

Configs are this:

# The maximum amount of heap to use, in MB. Default is 1000.
export HBASE_HEAPSIZE=16000

export HBASE_OPTS="$HBASE_OPTS -XX:MaxDirectMemorySize=16g"


Looking at UI, hardly any meta blocks in L1... a couple of hundred.

> Cell/DBB end-to-end on the read-path
> ------------------------------------
>                 Key: HBASE-11425
>                 URL: https://issues.apache.org/jira/browse/HBASE-11425
>             Project: HBase
>          Issue Type: Umbrella
>          Components: regionserver, Scanners
>    Affects Versions: 0.99.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>         Attachments: BenchmarkTestCode.zip, Benchmarks_Tests.docx, GC pics with evictions_4G
heap.png, HBASE-11425-E2E-NotComplete.patch, HBASE-11425.patch, Offheap reads in HBase using
BBs_V2.pdf, Offheap reads in HBase using BBs_final.pdf, gc.png, gets.png, heap.png, load.png,
median.png, ram.log
> Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the
entire read path, we can refer to this offheap buffer and avoid onheap copying.
> The high level items I can identify as of now are
> 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can
handle class by class)
> 2. Support Buffer based getter APIs in cell.  In read path we will create a new Cell
with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc.
> 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
> 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  (In read
> Will add subtasks under this.

This message was sent by Atlassian JIRA

View raw message