hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4720) Skip unnecessary NN operations in log aggregation
Date Tue, 23 Feb 2016 01:30:18 GMT
Ming Ma created YARN-4720:
-----------------------------

             Summary: Skip unnecessary NN operations in log aggregation
                 Key: YARN-4720
                 URL: https://issues.apache.org/jira/browse/YARN-4720
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Ming Ma


Log aggregation service could have unnecessary NN operations in the following scenarios:

* No new local log has been created since the last upload for the long running service scenario.
* NM uses {{ContainerLogAggregationPolicy}} that skips log aggregation for certain containers.

In the following code snippet, even though {{pendingContainerInThisCycle}} is empty, it still
creates the writer and then removes the file later. Thus it introduces unnecessary create/getfileinfo/delete
NN calls when NM doesn't aggregate logs for an app.
  
{noformat}
AppLogAggregatorImpl.java
......
        writer =
            new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
                this.userUgi);
......
      for (ContainerId container : pendingContainerInThisCycle) {
......
      }
......
            if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
              if (rename) {
                remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
              } else {
                remoteFS.delete(remoteNodeTmpLogFileForApp, false);
              }
            }
......
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message