hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-202) Log Aggregation generates a storm of fsync() for namenode
Date Tue, 06 Nov 2012 18:28:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491681#comment-13491681
] 

Robert Joseph Evans commented on YARN-202:
------------------------------------------

I think removing the flush is fine.  The file will get closed when the application finishes,
so the only issue is that if the NM crashes badly more logs may be lost then before. I am
+1. It a small change that reduces the load on the NN. I'll check it in.
                
> Log Aggregation generates a storm of fsync() for namenode
> ---------------------------------------------------------
>
>                 Key: YARN-202
>                 URL: https://issues.apache.org/jira/browse/YARN-202
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.0.2-alpha, 0.23.4
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: yarn-202.patch
>
>
> When the log aggregation is on, write to each aggregated container log causes hflush()
to be called. For large clusters, this can creates a lot of fsync() calls for namenode. 
> We have seen 6-7x increase in the average number of fsync operations compared to 1.0.x
on a large busy cluster. Over 99% of fsync ops were for log aggregation writing to tmp files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message