pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3507) It fails to run pig in local mode on a Kerberos enabled Hadoop cluster
Date Tue, 12 Nov 2013 20:06:17 GMT

    [ https://issues.apache.org/jira/browse/PIG-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820395#comment-13820395
] 

Rohini Palaniswamy commented on PIG-3507:
-----------------------------------------

This patch needs to be reverted if we are still supporting hadoop 0.23 as it will break compatibility.


http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/core/org/apache/hadoop/security/UserGroupInformation.java?view=markup
does not have setConfiguration method. 

Also I don't think the fix is correct. The problem that is happening is their core-site.xml
contains hadoop.security.authentication = kerberos and that is getting picked up as it is
loaded as a default resource in UserGroupInformation's configuration. We should fix the bin/pig
script to not load HADOOP_CONF_DIR in local mode. 

> It fails to run pig in local mode on a Kerberos enabled Hadoop cluster
> ----------------------------------------------------------------------
>
>                 Key: PIG-3507
>                 URL: https://issues.apache.org/jira/browse/PIG-3507
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.10.0, 0.11
>            Reporter: chiyang
>            Assignee: chiyang
>             Fix For: 0.13.0
>
>         Attachments: PIG-3507.patch
>
>
> It fails to run pig in local mode on a Kerberos enabled Hadoop cluster
> *Command*
> pig -x local <pig script>
> 		
> *Pig script*
> A = load '/etc/passwd';
> dump A;
> 	
> *Root cause*
> When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml
(hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However,
the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal()
is called from obtainTokensForNamenodes(), and causes the exception to happen.
> {noformat}
> public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration
conf) throws IOException {
>     if (!UserGroupInformation.isSecurityEnabled()) {
>         return;
>     }
>     obtainTokensForNamenodesInternal(credentials, ps, conf);
> }	
> {noformat}
> *Error*
> Pig Stack Trace
> ---------------
> ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494
More info to follow:
> Can't get JT Kerberos principal for use as renewer
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator
for alias A
> 	at org.apache.pig.PigServer.openIterator(PigServer.java:841)
> 	at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
> 	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
> 	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
> 	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
> 	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> 	at org.apache.pig.Main.run(Main.java:604)
> 	at org.apache.pig.Main.main(Main.java:157)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> 	at org.apache.pig.PigServer.storeEx(PigServer.java:940)
> 	at org.apache.pig.PigServer.store(PigServer.java:903)
> 	at org.apache.pig.PigServer.openIterator(PigServer.java:816)
> 	... 12 more
> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output Location Validation
Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
> Can't get JT Kerberos principal for use as renewer
> 	at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
> 	at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
> 	at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
> 	at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
> 	at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
> 	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
> 	at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
> 	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
> 	at org.apache.pig.PigServer.compilePp(PigServer.java:1327)
> 	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
> 	at org.apache.pig.PigServer.storeEx(PigServer.java:936)
> 	... 14 more
> Caused by: java.io.IOException: Can't get JT Kerberos principal for use as renewer
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:129)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
> 	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:127)
> 	at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
> 	... 24 more
> ================================================================================



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message