hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Leach (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14540) Write Ahead Log Batching Optimization
Date Fri, 02 Oct 2015 16:04:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941309#comment-14941309
] 

John Leach commented on HBASE-14540:
------------------------------------

Elliott, that is what I intuitively thought as well for a long time.  A few implementations
have changed my mind on this...

FYI, Here is a nice article on smart batching and why it is important even in low latency
systems.

http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html

Stack, let me know if I can help on the testing front.  I know you put a ton of work in on
the disruptor piece.

> Write Ahead Log Batching Optimization
> -------------------------------------
>
>                 Key: HBASE-14540
>                 URL: https://issues.apache.org/jira/browse/HBASE-14540
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: John Leach
>         Attachments: HBaseWALBlockingWaitStrategy.java
>
>
> The new write ahead log mechanism seems to batch too few mutations when running inside
the disruptor.  As we scaled our load up (many threads with small writes), we saw the number
of hdfs sync operations grow in concert with the number of writes.  Generally, one would expect
the size of the batches to grow but the number of actual sync operations to settle. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message