hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node
Date Fri, 14 Apr 2017 05:22:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968642#comment-15968642
] 

Jian He commented on YARN-6396:
-------------------------------

Hi [~zxu], 
only one thing is that if the valid permission becomes invalid maybe by accident at some point,
the verifyAndCreateRemoteLogDir won't be invoked any more to validate the permission, the
log-aggregation will eventually fail. 
I think it's a trade off between validation and efficiency. 
I'm ok with current approach. 
[~xgong], do you have comments ? 


> Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization
to decrease load for name node
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6396
>                 URL: https://issues.apache.org/jira/browse/YARN-6396
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: log-aggregation
>    Affects Versions: 3.0.0-alpha2
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>            Priority: Minor
>         Attachments: YARN-6396.000.patch
>
>
> Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization
to decrease load for name node.
> Currently for every application at each Node, verifyAndCreateRemoteLogDir will be called
before doing log aggregation, This will be a non trivial overhead for name node in a large
cluster since verifyAndCreateRemoteLogDir calls getFileStatus. Once the remote log directory
is created successfully, it is not necessary to call it again. It will be better to call verifyAndCreateRemoteLogDir
at LogAggregationService service initialization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message