hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shivram Mani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10012) Secure Oozie jobs fail with delegation token renewal exception in Namenode HA setup
Date Sun, 09 Feb 2014 17:57:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13895942#comment-13895942
] 

Shivram Mani commented on HADOOP-10012:
---------------------------------------

Once we ([~bhooshan] [~sumedh]) remove the patch we get the below stack trace
{code}
java.io.IOException: Failed to run job : Operation category WRITE is not supported in state
standby
	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:5333)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:388)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:859)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:43523)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)

	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:307)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:395)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:568)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:563)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:563)
	at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447)
	at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1355)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1139)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:446)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:456)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:712)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
	at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:261)
	at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:238)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
	at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:49)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Job Submission failed with exception 'java.io.IOException(Failed to run job : Operation category
WRITE is not supported in state standby
	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:5333)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:388)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:859)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:43523)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
Intercepting System.exit(1)
{code}
The above error (though related to failover), occurs when issuing the oozie hive action.

> Secure Oozie jobs fail with delegation token renewal exception in Namenode HA setup
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-10012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10012
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ha
>    Affects Versions: 2.1.1-beta
>            Reporter: Arpit Gupta
>            Assignee: Suresh Srinivas
>            Priority: Blocker
>             Fix For: 2.2.0
>
>         Attachments: HADOOP-10012.1.patch, HADOOP-10012.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message