hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-202) Log Aggregation generates a storm of fsync() for namenode
Date Tue, 06 Nov 2012 00:30:12 GMT
Kihwal Lee created YARN-202:

             Summary: Log Aggregation generates a storm of fsync() for namenode
                 Key: YARN-202
                 URL: https://issues.apache.org/jira/browse/YARN-202
             Project: Hadoop YARN
          Issue Type: Bug
    Affects Versions: 0.23.4, 2.0.2-alpha
            Reporter: Kihwal Lee
            Priority: Critical

When the log aggregation is on, write to each aggregated container log causes hflush() to
be called. For large clusters, this can creates a lot of fsync() calls for namenode. 

We have seen 6-7x increase in the average number of fsync operations compared to 1.0.x on
a large busy cluster. Over 99% of fsync ops were for log aggregation writing to tmp files.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message