hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
Date Thu, 15 Oct 2015 03:05:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958230#comment-14958230
] 

ramkrishna.s.vasudevan commented on HBASE-11425:
------------------------------------------------

bq.Have you fellas run this for a while? I seem to OOME easy enough. I tried with a small
heap, 4G, but it OOME'd then had to keep going back up to 16G again though I had a big offheap.
I tried to capture the increasing use of heap but this diagram is best I got... I've not done
heap analysis.. but in the diagram you can see heap use start to rise and then plummet...
now I am crawling doing Full GCs every couple of seconds.
Argh!! We have run some read related workloads for an hour or so but did not observe any OOMEs.
But our heap size was set at 32G. This is the case with YCSB and did not observe any full
GCs.
In case of PE tool we only used 9G heap size and 16G offheap size.  But did not observe any
fluctuations in the heap usage. 
bq.2015-10-14 16:51:10,058 DEBUG [main-BucketCacheWriter-2] bucket.BucketCache: This block
3f0157e7daee45fdb25202c496c95c46_1649898813 is still referred by 1 readers. Can not be freed
now
May be because of OOME some error handling is not done. Let us internally install a cluster
to test this. 

> Cell/DBB end-to-end on the read-path
> ------------------------------------
>
>                 Key: HBASE-11425
>                 URL: https://issues.apache.org/jira/browse/HBASE-11425
>             Project: HBase
>          Issue Type: Umbrella
>          Components: regionserver, Scanners
>    Affects Versions: 0.99.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>         Attachments: BenchmarkTestCode.zip, Benchmarks_Tests.docx, HBASE-11425-E2E-NotComplete.patch,
HBASE-11425.patch, Offheap reads in HBase using BBs_V2.pdf, Offheap reads in HBase using BBs_final.pdf,
gc.png, gets.png, heap.png, load.png, median.png
>
>
> Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the
entire read path, we can refer to this offheap buffer and avoid onheap copying.
> The high level items I can identify as of now are
> 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can
handle class by class)
> 2. Support Buffer based getter APIs in cell.  In read path we will create a new Cell
with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc.
> 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
> 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  (In read
path)
> Will add subtasks under this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message