hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surendra Singh Lilhore (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4783) Log aggregation failure for application when Nodemanager is restarted
Date Thu, 10 Mar 2016 05:40:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188714#comment-15188714
] 

Surendra Singh Lilhore commented on YARN-4783:
----------------------------------------------

Thanks [~jlowe] for the comment. 

Detail for the events from logs :
===================================

bq. 1. When the application completed
{noformat}
2014-03-01 19:11:03,527 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=yarn	OPERATION=Application Finished - Succeeded	TARGET=RMAppManager	RESULT=SUCCESS	APPID=application_xxxxxxxxxxxxxx_yyyyy
{noformat}

bq. 2. When the HDFS token was cancelled by the RM (check for 'Cancelling HDFS_DELEGATION_TOKEN
token 9 for yarn')
{noformat}
2014-03-01 19:21:12,347 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Token cancelation requested for identifier: HDFS_DELEGATION_TOKEN token 9 for yarn
{noformat}


bq. 3. When the nodemanager reconnected to the RM (and presumably started log aggregation
shortly afterwards)

NM Lost
{noformat}
2014-03-01 19:13:42,079 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
Deactivating Node host87:66666 as it is now LOST
{noformat}

NM Added
{noformat}
2014-03-02 08:44:31,785 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
host87:66666 Node Transitioned from NEW to RUNNING
{noformat}

{noformat}
2014-03-02 08:44:29,705 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
Failed to setup application log directory for application_xxxxxxxxxxxxxx_yyyyy
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 9 for yarn) can't be found in cache
	at org.apache.hadoop.ipc.Client.call(Client.java:1511)
	at org.apache.hadoop.ipc.Client.call(Client.java:1447)
{noformat}

> Log aggregation failure for application when Nodemanager is restarted 
> ----------------------------------------------------------------------
>
>                 Key: YARN-4783
>                 URL: https://issues.apache.org/jira/browse/YARN-4783
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.1
>            Reporter: Surendra Singh Lilhore
>
> Scenario :
> =========
> 1.Start NM with user dsperf:hadoop
> 2.Configure linux-execute user as dsperf
> 3.Submit application with yarn user 
> 4.Once few containers are allocated to NM 1
> 5.Nodemanager 1 is stopped  (wait for expiry )
> 6.Start node manager after application is completed
> 7.Check the log aggregation is happening for the containers log in NMLocal directory
> Expect Output :
> ===============
> Log aggregation should be succesfull
> Actual Output :
> ===============
> Log aggreation not successfull



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message