hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4720) Skip unnecessary NN operations in log aggregation
Date Tue, 23 Feb 2016 01:30:18 GMT
Ming Ma created YARN-4720:

             Summary: Skip unnecessary NN operations in log aggregation
                 Key: YARN-4720
                 URL: https://issues.apache.org/jira/browse/YARN-4720
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Ming Ma

Log aggregation service could have unnecessary NN operations in the following scenarios:

* No new local log has been created since the last upload for the long running service scenario.
* NM uses {{ContainerLogAggregationPolicy}} that skips log aggregation for certain containers.

In the following code snippet, even though {{pendingContainerInThisCycle}} is empty, it still
creates the writer and then removes the file later. Thus it introduces unnecessary create/getfileinfo/delete
NN calls when NM doesn't aggregate logs for an app.
        writer =
            new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
      for (ContainerId container : pendingContainerInThisCycle) {
            if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
              if (rename) {
                remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
              } else {
                remoteFS.delete(remoteNodeTmpLogFileForApp, false);

This message was sent by Atlassian JIRA

View raw message