hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicolas Liochon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
Date Fri, 29 Mar 2013 19:35:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617639#comment-13617639

Nicolas Liochon commented on HBASE-6295:

The table multiplexers has two drawbacks imho:
 - it's a new interface
 - it manages errors by dropping the puts that failed.
This is suitable for some applications, but not all of them.

I think it's possible to do something in the middle, and stick to the existing interface.
This would be:
 - we set two value for the write buffer: backgroundThreshold and blockingThreshold.
 - blockingThreshold works as today: we flush and block the callers until it's sent successfully
 - when backgroundThreshold is reached, we send data in the background, without blocking the
 - if we have too many errors we do as today: empty the buffer and raise the error to the
user at the next put.
 - flushCommit and puts works as today.
 - We keep the existing table behavior, and try, as today, to have not synchronization in
the HTable code.
Other comments:
 - if we want to be very efficient when merging calls to a given region server for different
tables, we should add this in the protobuf protocol.
 - if we change the messages sent to the region server, we could also let the region server
return a workload info. This workload would be used by the client to slow down their messages.
Some clients could have a 'nice' parameter to slow down more than others (typically map reduce
I will give a try to the send in background to see if it's possible to have something reasonable.

> Possible performance improvement in client batch operations: presplit and send in background
> --------------------------------------------------------------------------------------------
>                 Key: HBASE-6295
>                 URL: https://issues.apache.org/jira/browse/HBASE-6295
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client, Performance
>    Affects Versions: 0.96.0
>            Reporter: Nicolas Liochon
>              Labels: noob
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
>   add o to todolist
>   if todolist > maxsize or o last in list
>     split todolist per location
>     send split lists to region servers
>     clear todolist
>     wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is enough data
for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
>   get location
>   add o to todo location.todolist
>   if (location.todolist > maxLocationSize)
>     send location.todolist to region server 
>     clear location.todolist
>     // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be shared with
the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message