Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Tue, 6 Nov 2012 00:32:11 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <922372241.71645.1352161932050.JavaMail.jiratomcat@arcas>
In-Reply-To: <162795736.71643.1352161812521.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (YARN-202) Log Aggregation generates a storm of
 fsync() for namenode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491087#comment-13491087 ] 

Kihwal Lee commented on YARN-202:
---------------------------------

This problem will probably go away if we can leave out hflush() from LogWriter#append(). 
                
> Log Aggregation generates a storm of fsync() for namenode
> ---------------------------------------------------------
>
>                 Key: YARN-202
>                 URL: https://issues.apache.org/jira/browse/YARN-202
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.0.2-alpha, 0.23.4
>            Reporter: Kihwal Lee
>            Priority: Critical
>
> When the log aggregation is on, write to each aggregated container log causes hflush() to be called. For large clusters, this can creates a lot of fsync() calls for namenode. 
> We have seen 6-7x increase in the average number of fsync operations compared to 1.0.x on a large busy cluster. Over 99% of fsync ops were for log aggregation writing to tmp files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira