hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11425) Cell/DBB end-to-end on the read-path
Date Thu, 15 Oct 2015 06:29:07 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

ramkrishna.s.vasudevan updated HBASE-11425:
    Attachment: GC pics with evictions_4G heap.png

Tried out the experiments in a new cluster with single node. 
Loaded around 15G of data with 10 regions. 
Initially configured 4G as heap space, offheap space as 5G and the bucket cache size as 4G.
Ran a pure read workload c for 30 mins.  With 50 threads.  I am not running into OOME and
also the block eviction part is fine.  The bigger GCs are around 450ms to 500ms.  Repeated
the same experiment with 10 G of heap space also.  With this configuration we are sure that
evictions are happening from the bucket cache. Attaching a GC snapshot for 5 mins captured
during the workload test. 
Also in your experiment I think your data does not fit into the bucket cache and hence it
is trying to evict. Or if it is fitting into the bucket cache, probably that was a file that
was trying to get compacted and there was a reader referencing to it and due to the OOME the
ref count decrement did not happen and the forceful eviction was failing. Will keep checking
this.  Any logs can you attach when this happened?  Easier to debug (hopefully).

> Cell/DBB end-to-end on the read-path
> ------------------------------------
>                 Key: HBASE-11425
>                 URL: https://issues.apache.org/jira/browse/HBASE-11425
>             Project: HBase
>          Issue Type: Umbrella
>          Components: regionserver, Scanners
>    Affects Versions: 0.99.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>         Attachments: BenchmarkTestCode.zip, Benchmarks_Tests.docx, GC pics with evictions_4G
heap.png, HBASE-11425-E2E-NotComplete.patch, HBASE-11425.patch, Offheap reads in HBase using
BBs_V2.pdf, Offheap reads in HBase using BBs_final.pdf, gc.png, gets.png, heap.png, load.png,
> Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the
entire read path, we can refer to this offheap buffer and avoid onheap copying.
> The high level items I can identify as of now are
> 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can
handle class by class)
> 2. Support Buffer based getter APIs in cell.  In read path we will create a new Cell
with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc.
> 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
> 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  (In read
> Will add subtasks under this.

This message was sent by Atlassian JIRA

View raw message