accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billie Rinaldi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1083) add concurrency to HDFS write-ahead log
Date Fri, 08 Mar 2013 16:16:13 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597249#comment-13597249
] 

Billie Rinaldi commented on ACCUMULO-1083:
------------------------------------------

It looks like the permissions are configured so that administrators can edit anyone's comment,
and no one else is allowed to edit comments.  This was copied from Hadoop's setup.  The system
makes it really difficult to have custom permissions -- administrators can't change the permissions,
and when you ask INFRA to do it typically they want to just copy some existing project's setup.
 If people think it's important for everyone to be able to edit their own comments, we can
ask.
                
> add concurrency to HDFS write-ahead log
> ---------------------------------------
>
>                 Key: ACCUMULO-1083
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1083
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Adam Fuchs
>             Fix For: 1.6.0
>
>         Attachments: walog-performance.jpg
>
>
> When running tablet servers on beefy nodes (lots of disks), the write-ahead log can be
a serious bottleneck. Today we ran a test of 1.5-SNAPSHOT on an 8-node (plus a master node)
cluster in which the nodes had 32 cores and 15 drives each. Running with write-ahead log off
resulted in a >4x performance improvement sustained over a long period.
> I believe the culprit is that the WAL is only using one file at a time per tablet server,
which means HDFS is only appending to one drive (plus replicas). If we increase the number
of concurrent WAL files supported on a tablet server we could probably drastically improve
the performance on systems with many disks. As it stands, I believe Accumulo is significantly
more optimized for a larger number of smaller nodes (3-4 drives).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message