hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4755) Optimize sending appACLsUpdated event to TimelineServer while recovering completed applications
Date Wed, 02 Mar 2016 14:44:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175693#comment-15175693

Naganarasimha G R commented on YARN-4755:

I think we discussed these topics in the following JIRAs' YARN-3127 and YARN-4392 and the
conclusion was, we were ok with republishing the events with exact data rather than not publishing
at all because its not guaranteed that ATS events for apps in state store are successfully
To actually see *appACLsUpdated* need not be separately published again we can directly publish
this operation along with appCreatedEvent thus avoids one additional entity processing. Need
to check what would be the ideal place to have acl information and also ensure its compatible
with the current code.

> Optimize sending appACLsUpdated event to TimelineServer while recovering completed applications
> -----------------------------------------------------------------------------------------------
>                 Key: YARN-4755
>                 URL: https://issues.apache.org/jira/browse/YARN-4755
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Rohith Sharma K S
>            Assignee: Naganarasimha G R
> In method {{RMAppManager#createAndPopulateNewRMApp}}, appACLsUpdated is sent to timelineserver
for every application that get created. 
> {code}
>  private RMAppImpl createAndPopulateNewRMApp(
>       ApplicationSubmissionContext submissionContext, long submitTime,
>       String user, boolean isRecovery) throws YarnException {
> 		//
>                 //
>     String appViewACLs = submissionContext.getAMContainerSpec()
>         .getApplicationACLs().get(ApplicationAccessType.VIEW_APP);
>     rmContext.getSystemMetricsPublisher().appACLsUpdated(
>         application, appViewACLs, System.currentTimeMillis());
>     return application;
>   }
> {code}
> Say if we have 10K completed applications to recover, 30K events will be generated i.e
app_created, app_finished and app_acl_updated. For completed applications, I think need not
to send app-acl-updated event with which gradually reduce load on the dispatcher. 
> Eventhough MultiDispatcher is used to publish timeline events, it is bottle neck when
max-completed is configured very high value may be 100K.

This message was sent by Atlassian JIRA

View raw message