hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
Date Thu, 04 Feb 2016 06:45:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131831#comment-15131831
] 

ramkrishna.s.vasudevan commented on HBASE-15204:
------------------------------------------------

Thanks for the reviews. As I said this was only a first cut to show the change. 
Passing cellCount to batchMutate is not neat I agree. Let me see what to do. 
But one thing - this change is only for Clients that work with CellBlock and not for PB based
clients. Still I would say this is better if we can get the estimate. Let me check for the
retry failures case. But some how if we can get an estimate - even if it is slighly bigger
we can reduce the count of the garbage. 
Changing the default size of the Arraylist in WAL is another way which I too thought but what
to set as the initial value was not clear to me. Let me see if we can do something better.
If not will leave this for now. 

> Try to estimate the cell count for adding into WALEdit
> ------------------------------------------------------
>
>                 Key: HBASE-15204
>                 URL: https://issues.apache.org/jira/browse/HBASE-15204
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>
>         Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg
>
>
> The write path profiling shows that when we try to add Cells to WALEdits we try to do
a lot of Array copy inorder to grow the Arraylist backing the WALEdits. In a simple one min
profiling of the write path with 50 YCSB threads shows around 261MB of allocation done for
the Array copy to happen. We can try to avoid that. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message