hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2256) Too many nodemanager and resourcemanager audit logs are generated
Date Mon, 08 Sep 2014 21:20:29 GMT

    [ https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126119#comment-14126119
] 

Varun Saxena commented on YARN-2256:
------------------------------------

bq. Someone correct me if I'm wrong, but I'm fairly certain that the intent of this information
is to be the equivalent of the HDFS audit log. In other words, setting these to debug completely
defeats the purpose. Instead, I suspect the real culprit is that the log4j settings are wrong
for the node manager process.

[~aw], the issue raised was basically for both NM and RM. I have updated the description to
reflect that. The issue here is that some of the container related operations' audit logs
in both NM and RM are too frequent and too many. This may impact performance as well.

Now, there are 2 solutions possible, either remove these logs or change the log level, so
that they do not occur in live environment and can be opened only when required. 
As I wasnt sure if these audit logs have to be removed or not, I changed the log level for
some of these logs in RM and all of them in NM. To ensure this, I supported printing of audit
logs at different levels, as is done in HBase (as per my info). This is handled as part of
YARN-2287

Now for NM, you are correct, Log level can be changed in log4j properties to suppress these
logs if required. But for RM, as not all logs have to be suppressed, this cant be done. So
to be consistent, I added log levels for both NM and RM. 

If its agreeable to remove these audit logs, that can be a possible solution as well. Pls
suggest. 

> Too many nodemanager and resourcemanager audit logs are generated
> -----------------------------------------------------------------
>
>                 Key: YARN-2256
>                 URL: https://issues.apache.org/jira/browse/YARN-2256
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager, resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-2256.patch
>
>
> Following audit logs are generated too many times(due to the possibility of a large number
of containers) :
> 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a container
> 2. In RM - Audit logs corresponding to AM allocating a container and AM releasing a container
> We can have different log levels even for NM and RM audit logs and move these successful
container related logs to DEBUG.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message