hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
Date Mon, 29 Aug 2016 22:28:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447251#comment-15447251

Jason Lowe commented on YARN-5549:

Totally agree with Daniel that this is only useful for debugging a live issue.  However there
is a logLevel servlet on all the major daemons which can dynamically change the log level
of user-specified loggers, so this can be debugged live without too much hassle on the RM
side assuming the issue is repeatable and the user simply needs to re-run it after the log
level has been updated.

As for the other ways to debug it, I agree the NM provides the most details by far.  However
it can be a real pain on a sizeable cluster to update all NMs for this and it affects all
containers on that node rather than the one we're interested in.  And if we don't update all
the nodes it's hard to know which NM will be used for the next test of the AM.  That leaves
the job.xml approach, but YARN doesn't require a job.xml for applications.  So if we're debugging
a custom YARN application that could be challenging if both the RM and NM aren't providing
any details as to what command is being launched, the app framework isn't helping either,
and the logs from the failed command also aren't sufficient to know what happened.

I'd like to get rid of the log as long as there's a reasonable alternative to getting the
information elsewhere when debugging weird launch failures.  Maybe we need a more explicit
way for users to debug their own app launch failures?  That would be far preferable than having
to involve a cluster admin every time.  For example there could be a flag in the submission
context that causes the NM to log the command line and/or startup script to a separate log
file in the YARN log directory for the container.  Obviously for another JIRA.  Just wanted
to point out that updating the log level for live debugging is not something that requires
a restart.

> AMLauncher.createAMContainerLaunchContext() should not log the command to be launched
> ------------------------------------------------------------------------------------------------------
>                 Key: YARN-5549
>                 URL: https://issues.apache.org/jira/browse/YARN-5549
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.7.2
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>            Priority: Critical
>         Attachments: YARN-5549.001.patch, YARN-5549.002.patch, YARN-5549.003.patch, YARN-5549.004.patch
> The command could contain sensitive information, such as keystore passwords or AWS credentials
or other.  Instead of logging it as INFO, we should log it as DEBUG and include a property
to disable logging it at all.  Logging it to a different logger would also be viable and may
create a smaller administrative footprint.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message