accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1083) add concurrency to HDFS write-ahead log
Date Fri, 22 Feb 2013 11:58:12 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584219#comment-13584219
] 

Eric Newton commented on ACCUMULO-1083:
---------------------------------------

Concurrent, as in, a sub-set of tablets use a log, or concurrent, as in, there are multiple
log files available to log to, as described in the Big Table paper?

Note that in 1.5, default replication of the WAL is probably 3, whereas it used to be 2.

                
> add concurrency to HDFS write-ahead log
> ---------------------------------------
>
>                 Key: ACCUMULO-1083
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1083
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: logger, tserver
>            Reporter: Adam Fuchs
>
> When running tablet servers on beefy nodes (lots of disks), the write-ahead log can be
a serious bottleneck. Today we ran a test of 1.5-SNAPSHOT on an 8-node (plus a master node)
cluster in which the nodes had 32 cores and 15 drives each. Running with write-ahead log off
resulted in a >4x performance improvement sustained over a long period.
> I believe the culprit is that the WAL is only using one file at a time per tablet server,
which means HDFS is only appending to one drive (plus replicas). If we increase the number
of concurrent WAL files supported on a tablet server we could probably drastically improve
the performance on systems with many disks. As it stands, I believe Accumulo is significantly
more optimized for a larger number of smaller nodes (3-4 drives).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message