hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-162) nodemanager log aggregation has scaling issues with namenode
Date Fri, 16 Nov 2012 23:13:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499232#comment-13499232
] 

Siddharth Seth commented on YARN-162:
-------------------------------------

The patch does change behaviour. Prior to this, directory creation failure would cause the
app to be marked as FAILED on the NM. Failing to aggregate logs for some containers would
not fail the app though. With the patch, dir creation failing does not fail the app.
                
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
>                 Key: YARN-162
>                 URL: https://issues.apache.org/jira/browse/YARN-162
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 0.23.3
>            Reporter: Nathan Roberts
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Critical
>         Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust
FDs to the point where datanodes can't check-in.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message