hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
Date Thu, 29 Jan 2015 19:45:37 GMT
Jack Chen created YARN-3112:
-------------------------------

             Summary: AM restart and keep containers from previous attempts, then new container
launch failed
                 Key: YARN-3112
                 URL: https://issues.apache.org/jira/browse/YARN-3112
             Project: Hadoop YARN
          Issue Type: Bug
          Components: applications, resourcemanager
    Affects Versions: 2.6.0
         Environment: in real linux cluster
            Reporter: Jack Chen


This error is very similar to YARN-713, 1839, but i have check the solution of those jira,
the patches are already included in my version. I think this error is caused by the different
NMTokens between old and new appattempts. New AM has inherited the old tokens from previous
AM according to my configuration (keepContainers=true), so the token for new containers are
replaced by the old one in the NMTokenCache.

206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
Container launch failed for      container_1422546145900_0001_02_000002 : org.apache.hadoop.security.token.SecretManager$InvalidToken:
No NMToken sent for ixk02:47625
 207 ›   at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt
    ocolProxy.java:256)
 208 ›   at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc
    olProxy.java:246)
 209 ›   at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
 210 ›   at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
 211 ›   at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
 212 ›   at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
 213 ›   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 214 ›   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 215 ›   at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message