accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Fuchs (JIRA)" <>
Subject [jira] [Created] (ACCUMULO-1083) add concurrency to HDFS write-ahead log
Date Fri, 22 Feb 2013 02:54:12 GMT
Adam Fuchs created ACCUMULO-1083:

             Summary: add concurrency to HDFS write-ahead log
                 Key: ACCUMULO-1083
             Project: Accumulo
          Issue Type: Improvement
          Components: logger, tserver
            Reporter: Adam Fuchs

When running tablet servers on beefy nodes (lots of disks), the write-ahead log can be a serious
bottleneck. Today we ran a test of 1.5-SNAPSHOT on an 8-node (plus a master node) cluster
in which the nodes had 32 cores and 15 drives each. Running with write-ahead log off resulted
in a >4x performance improvement sustained over a long period.

I believe the culprit is that the WAL is only using one file at a time per tablet server,
which means HDFS is only appending to one drive (plus replicas). If we increase the number
of concurrent WAL files supported on a tablet server we could probably drastically improve
the performance on systems with many disks. As it stands, I believe Accumulo is significantly
more optimized for a larger number of smaller nodes (3-4 drives).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message