hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
Date Sat, 06 Feb 2016 07:03:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135646#comment-15135646

ramkrishna.s.vasudevan commented on HBASE-15204:

bq.the return is orthogonal to why we call the CP. We are now dependent on CP implementation
to do work for us.
[~saint.ack@gmail.com] Not very clear on this part. If CP is returning some mutations - we
explicitly add that to a WALEdit. I am just counting if something is there. That is only to
get an exact count because that will also be added to the WALEdit that we create in doMiniBatchMutation.
bq.It is kind of odd that doMiniBatch takes a cell count when the other param is a fat batchOp
with description of what the edit its. If cell count should be anywhere, it should be in batchop?
I can remove that if we don't want that. Ya in batchOp we could add the Cellcount but many
of the mutations may not be applied in that batch. But if we don't want the cellCountFromCP
to be passed then am fine with it too. I thought since we can use it and get the exact count
it is preferable to use it. 
bq.Do we have to add a Cell at a time? Can we not pass the whole list in one go?
LEt me check this out. Since WALEdit was backed by an Arraylist I thought if we can size that
backing list properly we are fine with it.
bq.I appreciate the savings in garbage but perhaps we can do it in a cleaner way.
Okie. I can revert this patch and see what else can be done.

> Try to estimate the cell count for adding into WALEdit
> ------------------------------------------------------
>                 Key: HBASE-15204
>                 URL: https://issues.apache.org/jira/browse/HBASE-15204
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0, 1.3.0
>         Attachments: HBASE-15204.patch, HBASE-15204_1.patch, HBASE-15204_1.patch, WAlEdit_add_allocation.jpg,
> The write path profiling shows that when we try to add Cells to WALEdits we try to do
a lot of Array copy inorder to grow the Arraylist backing the WALEdits. In a simple one min
profiling of the write path with 50 YCSB threads shows around 261MB of allocation done for
the Array copy to happen. We can try to avoid that. 

This message was sent by Atlassian JIRA

View raw message