hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fengyun RAO <raofeng...@gmail.com>
Subject YarnException: Unauthorized request to start container. This token is expired.
Date Sun, 23 Mar 2014 09:37:57 GMT
What does this exception mean? I googled a lot, all the results tell me
it's because the time is not synchronized between datanode and namenode.
However, I checked all the servers, that the ntpd service is on, and the
time differences are less than 1 second.
What's more, the tasks are not always failing on certain datanodes.
It fails and then it restarts and succeeds. If it were the time problem, I
guess it would always fail.

My hadoop version is CDH5 beta. Below is the detailed log:

14/03/23 14:57:06 INFO mapreduce.Job: Running job: job_1394434496930_0032
14/03/23 14:57:17 INFO mapreduce.Job: Job job_1394434496930_0032 running in
uber mode : false
14/03/23 14:57:17 INFO mapreduce.Job:  map 0% reduce 0%
14/03/23 15:08:01 INFO mapreduce.Job: Task Id :
attempt_1394434496930_0032_m_000034_0, Status : FAILED
Container launch failed for container_1394434496930_0032_01_000041 :
org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to
start container.
This token is expired. current time is 1395558481146 found 1395558443384
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
        at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:370)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

14/03/23 15:08:02 INFO mapreduce.Job:  map 1% reduce 0%
14/03/23 15:09:36 INFO mapreduce.Job: Task Id :
attempt_1394434496930_0032_m_000036_0, Status : FAILED
Container launch failed for container_1394434496930_0032_01_000038 :
org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to
start container.
This token is expired. current time is 1395558575889 found 1395558443245
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
        at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:370)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

Mime
View raw message