hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1751) Improve MiniYarnCluster and LogCLIHelpers for log aggregation testing
Date Sat, 10 May 2014 21:55:28 GMT

    [ https://issues.apache.org/jira/browse/YARN-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993565#comment-13993565

Jason Lowe commented on YARN-1751:

Despite them both being small changes, I think these should be separate JIRA since they're
otherwise unrelated changes for different problems and can stand on their own.  We can morph
this JIRA into one of them and file a new one to cover the other.

For the LogCLIHelpers change, I think it should be calling FileContext.getFileContext(remoteAppLogDir.toUri(),
conf) in case the remoteAppLogDir is not on the default filesystem.  There's also the question
of whether it should guard against a null conf, since oddly despite LogCLIHelpers being Configurable
it isn't using the config until after this change.  I think I'm leaning towards leaving it
null and letting the NPE occur so callers will fix it.  We've had lots of performance problems
and other weirdness in the past when code forgot to pass down a custom config and things sorta
worked with the default one.

+1 for the MiniYarnCluster change.

> Improve MiniYarnCluster and LogCLIHelpers for log aggregation testing
> ---------------------------------------------------------------------
>                 Key: YARN-1751
>                 URL: https://issues.apache.org/jira/browse/YARN-1751
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: YARN-1751-trunk.patch
> MiniYarnCluster specifies individual remote log aggregation root dir for each NM. Test
code that uses MiniYarnCluster won't be able to get the value of log aggregation root dir.
The following code isn't necessary in MiniYarnCluster.
>       File remoteLogDir =
>           new File(testWorkDir, MiniYARNCluster.this.getName()
>               + "-remoteLogDir-nm-" + index);
>       remoteLogDir.mkdir();
>       config.set(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
>           remoteLogDir.getAbsolutePath());
> In LogCLIHelpers.java, dumpAllContainersLogs should pass its conf object to FileContext.getFileContext()

This message was sent by Atlassian JIRA

View raw message