hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingjie Lai (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3083) HA+security: failed to run a mapred job from yarn after a manual failover
Date Tue, 13 Mar 2012 07:05:38 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mingjie Lai updated HDFS-3083:
------------------------------

    Description: 
Steps to reproduce:
- turned on ha and security
- run a mapred job, and wait to finish
- failover to another namenode
- run the mapred job again, it fails. 

Checking the job delegation token, it still indicates the original active namenode, and causes
nm failed to obtain a dt for the new active nn. (?) 

{code}
$ hdfs dfs -cat hdfs://ns1:8020/tmp/hadoop-yarn/staging/yarn/.staging/job_1331619043691_0001/appTokens
HDTS
     ha-hdfs:ns1@(yarn/nn1.hadoop.local@HADOOP.LOCALDOMAINyarn�6
�L��6.�ЛFs��r�%�B�'��{pR�HDFS_DELEGATION_TOKEN
                                                    ha-hdfs:ns
{code}

Exceptions:
{code}
12/03/13 06:19:44 INFO mapred.ResourceMgrDelegate: Submitted application application_1331619043691_0002
to ResourceManager at nn1.hadoop.local/10.177.23.38:7090
12/03/13 06:19:45 INFO mapreduce.Job: The url to track the job: http://nn1.hadoop.local:7050/proxy/application_1331619043691_0002/
12/03/13 06:19:45 INFO mapreduce.Job: Running job: job_1331619043691_0002
12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 running in uber mode : false
12/03/13 06:19:47 INFO mapreduce.Job:  map 0% reduce 0%
12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 failed with state FAILED
due to: Application application_1331619043691_0002 failed 1 times due to AM Container for
appattempt_1331619043691_0002_000001 exited with  exitCode: -1000 due to: RemoteTrace: 
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN
token 40 for yarn) can't be found in cache
	at org.apache.hadoop.ipc.Client.call(Client.java:1159)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
	at $Proxy28.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:622)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
	at $Proxy29.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1260)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:718)
	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:88)
	at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49)
	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157)
	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
 at LocalTrace: 
	org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: token (HDFS_DELEGATION_TOKEN
token 40 for yarn) can't be found in cache
	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217)
	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.update(ResourceLocalizationService.java:827)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:497)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:222)
	at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:46)
	at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:57)
	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:355)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1660)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1656)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1654)

.Failing this attempt.. Failing the application.
12/03/13 06:19:47 INFO mapreduce.Job: Counters: 0
Job ended: Tue Mar 13 06:19:47 UTC 2012
The job took 3 seconds.
{code}


  was:
Steps to reproduce:
- turned on ha and security
- run a mapred job, and wait to finish
- failover to another namenode
- run the mapred job again, it fails. 

Checking the job delegation token, it's still indicate the original active namenode. It causes
nm failed to obtain a dt for the new nn. (?) 

{code}
$ hdfs dfs -cat hdfs://ns1:8020/tmp/hadoop-yarn/staging/yarn/.staging/job_1331619043691_0001/appTokens
HDTS
     ha-hdfs:ns1@(yarn/nn1.hadoop.local@HADOOP.LOCALDOMAINyarn�6
�L��6.�ЛFs��r�%�B�'��{pR�HDFS_DELEGATION_TOKEN
                                                    ha-hdfs:ns
{code}

Exceptions:
{code}
12/03/13 06:19:44 INFO mapred.ResourceMgrDelegate: Submitted application application_1331619043691_0002
to ResourceManager at nn1.hadoop.local/10.177.23.38:7090
12/03/13 06:19:45 INFO mapreduce.Job: The url to track the job: http://nn1.hadoop.local:7050/proxy/application_1331619043691_0002/
12/03/13 06:19:45 INFO mapreduce.Job: Running job: job_1331619043691_0002
12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 running in uber mode : false
12/03/13 06:19:47 INFO mapreduce.Job:  map 0% reduce 0%
12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 failed with state FAILED
due to: Application application_1331619043691_0002 failed 1 times due to AM Container for
appattempt_1331619043691_0002_000001 exited with  exitCode: -1000 due to: RemoteTrace: 
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN
token 40 for yarn) can't be found in cache
	at org.apache.hadoop.ipc.Client.call(Client.java:1159)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
	at $Proxy28.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:622)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
	at $Proxy29.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1260)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:718)
	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:88)
	at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49)
	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157)
	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
 at LocalTrace: 
	org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: token (HDFS_DELEGATION_TOKEN
token 40 for yarn) can't be found in cache
	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217)
	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.update(ResourceLocalizationService.java:827)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:497)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:222)
	at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:46)
	at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:57)
	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:355)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1660)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1656)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1654)

.Failing this attempt.. Failing the application.
12/03/13 06:19:47 INFO mapreduce.Job: Counters: 0
Job ended: Tue Mar 13 06:19:47 UTC 2012
The job took 3 seconds.
{code}


    
> HA+security: failed to run a mapred job from yarn after a manual failover
> -------------------------------------------------------------------------
>
>                 Key: HDFS-3083
>                 URL: https://issues.apache.org/jira/browse/HDFS-3083
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha, security
>    Affects Versions: 0.24.0, 0.23.3
>            Reporter: Mingjie Lai
>            Priority: Critical
>             Fix For: 0.24.0, 0.23.3
>
>
> Steps to reproduce:
> - turned on ha and security
> - run a mapred job, and wait to finish
> - failover to another namenode
> - run the mapred job again, it fails. 
> Checking the job delegation token, it still indicates the original active namenode, and
causes nm failed to obtain a dt for the new active nn. (?) 
> {code}
> $ hdfs dfs -cat hdfs://ns1:8020/tmp/hadoop-yarn/staging/yarn/.staging/job_1331619043691_0001/appTokens
> HDTS
>      ha-hdfs:ns1@(yarn/nn1.hadoop.local@HADOOP.LOCALDOMAINyarn�6
> �L��6.�ЛFs��r�%�B�'��{pR�HDFS_DELEGATION_TOKEN
>                                                     ha-hdfs:ns
> {code}
> Exceptions:
> {code}
> 12/03/13 06:19:44 INFO mapred.ResourceMgrDelegate: Submitted application application_1331619043691_0002
to ResourceManager at nn1.hadoop.local/10.177.23.38:7090
> 12/03/13 06:19:45 INFO mapreduce.Job: The url to track the job: http://nn1.hadoop.local:7050/proxy/application_1331619043691_0002/
> 12/03/13 06:19:45 INFO mapreduce.Job: Running job: job_1331619043691_0002
> 12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 running in uber mode
: false
> 12/03/13 06:19:47 INFO mapreduce.Job:  map 0% reduce 0%
> 12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 failed with state FAILED
due to: Application application_1331619043691_0002 failed 1 times due to AM Container for
appattempt_1331619043691_0002_000001 exited with  exitCode: -1000 due to: RemoteTrace: 
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN
token 40 for yarn) can't be found in cache
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1159)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
> 	at $Proxy28.getFileInfo(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:622)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> 	at $Proxy29.getFileInfo(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1260)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:718)
> 	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:88)
> 	at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49)
> 	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157)
> 	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> 	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153)
> 	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
>  at LocalTrace: 
> 	org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: token (HDFS_DELEGATION_TOKEN
token 40 for yarn) can't be found in cache
> 	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217)
> 	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.update(ResourceLocalizationService.java:827)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:497)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:222)
> 	at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:46)
> 	at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:57)
> 	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:355)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1660)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1656)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1654)
> .Failing this attempt.. Failing the application.
> 12/03/13 06:19:47 INFO mapreduce.Job: Counters: 0
> Job ended: Tue Mar 13 06:19:47 UTC 2012
> The job took 3 seconds.
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message