hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11425) Cell/DBB end-to-end on the read-path
Date Thu, 15 Oct 2015 00:01:18 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HBASE-11425:
    Attachment: heap.png

Have you fellas run this for a while? I seem to OOME easy enough. I tried with a small heap,
4G, but it OOME'd then had to keep going back up to 16G again though I had a big offheap.
 I tried to capture the increasing use of heap but this diagram is best I got... I've not
done heap analysis.. but in the diagram you can see heap use start to rise and then plummet...
now I am crawling doing Full GCs every couple of seconds.

The last meaningful log was this in regionserver log:

2015-10-14 16:51:10,058 DEBUG [main-BucketCacheWriter-2] bucket.BucketCache: This block 3f0157e7daee45fdb25202c496c95c46_1649898813
is still referred by 1 readers. Can not be freed now

It started complaining 50minutes ago...

> Cell/DBB end-to-end on the read-path
> ------------------------------------
>                 Key: HBASE-11425
>                 URL: https://issues.apache.org/jira/browse/HBASE-11425
>             Project: HBase
>          Issue Type: Umbrella
>          Components: regionserver, Scanners
>    Affects Versions: 0.99.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>         Attachments: BenchmarkTestCode.zip, Benchmarks_Tests.docx, HBASE-11425-E2E-NotComplete.patch,
HBASE-11425.patch, Offheap reads in HBase using BBs_V2.pdf, Offheap reads in HBase using BBs_final.pdf,
gc.png, gets.png, heap.png, load.png, median.png
> Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the
entire read path, we can refer to this offheap buffer and avoid onheap copying.
> The high level items I can identify as of now are
> 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can
handle class by class)
> 2. Support Buffer based getter APIs in cell.  In read path we will create a new Cell
with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc.
> 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
> 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  (In read
> Will add subtasks under this.

This message was sent by Atlassian JIRA

View raw message