hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: Oozie workflow error - renewing token issue
Date Wed, 30 Jan 2013 20:14:33 GMT
Cobert,

[Moving thread to user@oozie.a.o, BCCing common-user@hadoop.a.o]

* What version of Oozie are you using?
* Is the cluster a secure setup (Kerberos enabled)?
* Would you mind posting the complete launcher logs?

Thx



On Wed, Jan 30, 2013 at 6:14 AM, Corbett Martin <comartin@nhin.com> wrote:

> Thanks for the tip.
>
> The sqoop command listed in the stdout log file is:
>                     sqoop
>                     import
>                     --driver
>                     org.apache.derby.jdbc.ClientDriver
>                     --connect
>                     jdbc:derby://test-server:1527/mondb
>                     --username
>                     monuser
>                     --password
>                     xxxxxxxxx
>                     --table
>                     MONITOR
>                     --split-by
>                     request_id
>                     --target-dir
>                     /mon/import
>                     --append
>                     --incremental
>                     append
>                     --check-column
>                     request_id
>                     --last-value
>                     2000000
>
> The following information is from the sterr and stdout log files.
>
> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0
> # more stderr
> No such sqoop tool: sqoop. See 'sqoop help'.
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
>
>
> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0
> # more stdout
> ...
> ...
> ...
> >>> Invoking Sqoop command line now >>>
>
> 1598 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has
> not been set in the environment. Cannot check for additional configuration.
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
>
> Oozie Launcher failed, finishing Hadoop job gracefully
>
> Oozie Launcher ends
>
>
> ~Corbett Martin
> Software Architect
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Tuesday, January 29, 2013 10:11 PM
> To: <user@hadoop.apache.org>
> Subject: Re: Oozie workflow error - renewing token issue
>
> The job that Oozie launches for your action, which you are observing is
> failing, does its own logs (task logs) show any errors?
>
> On Wed, Jan 30, 2013 at 4:59 AM, Corbett Martin <comartin@nhin.com> wrote:
> > Oozie question
> >
> >
> >
> > I'm trying to run an Oozie workflow (sqoop action) from the Hue
> > console and it fails every time.  No exception in the oozie log but I
> > see this in the Job Tracker log file.
> >
> >
> >
> > Two primary issues seem to be
> >
> > 1.   Client mapred tries to renew a token with renewer specified as mr
> token
> >
> >
> >
> > And
> >
> >
> >
> > 2.   Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
> >
> >
> >
> > Any ideas how to get past this?
> >
> >
> >
> > Full Stacktrace:
> >
> >
> >
> > 2013-01-29 17:11:28,860 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Creating password for identifier: owner=hdfs, renewer=mr token,
> > realUser=oozie, issueDate=1359501088860, maxDate=1360105888860,
> > sequenceNumber=75, masterKeyId=8
> >
> > 2013-01-29 17:11:28,871 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Creating password for identifier: owner=hdfs, renewer=mr token,
> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
> > sequenceNumber=76, masterKeyId=8
> >
> > 2013-01-29 17:11:29,202 INFO
> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
> > registering token for renewal for service =10.204.12.62:8021 and jobID
> > =
> > job_201301231648_0029
> >
> > 2013-01-29 17:11:29,211 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Token renewal requested for identifier: owner=hdfs, renewer=mr token,
> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
> > sequenceNumber=76, masterKeyId=8
> >
> > 2013-01-29 17:11:29,211 ERROR
> > org.apache.hadoop.security.UserGroupInformation:
> > PriviledgedActionException as:mapred (auth:SIMPLE)
> > cause:org.apache.hadoop.security.AccessControlException: Client mapred
> > tries to renew a token with renewer specified as mr token
> >
> > 2013-01-29 17:11:29,211 WARN org.apache.hadoop.security.token.Token:
> > Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
> >
> > 2013-01-29 17:11:29,211 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 8021, call renewDelegationToken(Kind:
> > MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, Ident: 00 04
> > 68 64
> > 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 3c 88 94 58
> > 67 8a
> > 01 3c ac a0 dc 67 4c 08), rpc version=2, client version=28,
> > methodsFingerPrint=1830206421 from 10.204.12.62:9706: error:
> > org.apache.hadoop.security.AccessControlException: Client mapred tries
> > to renew a token with renewer specified as mr token
> >
> > org.apache.hadoop.security.AccessControlException: Client mapred tries
> > to renew a token with renewer specified as mr token
> >
> >           at
> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec
> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
> >
> >           at
> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja
> > va:3738)
> >
> >           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> >           at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:39)
> >
> >           at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:25)
> >
> >           at java.lang.reflect.Method.invoke(Method.java:597)
> >
> >           at
> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call
> > (WritableRpcEngine.java:474)
> >
> >           at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
> >
> >           at java.security.AccessController.doPrivileged(Native
> > Method)
> >
> >           at javax.security.auth.Subject.doAs(Subject.java:396)
> >
> >           at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> > ion.java:1332)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
> >
> > 2013-01-29 17:11:29,212 ERROR
> > org.apache.hadoop.security.UserGroupInformation:
> > PriviledgedActionException as:mapred (auth:SIMPLE)
> >
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> > Client mapred tries to renew a token with renewer specified as mr
> > token
> >
> >           at
> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec
> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
> >
> >           at
> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja
> > va:3738)
> >
> >           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> >           at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:39)
> >
> >           at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:25)
> >
> >           at java.lang.reflect.Method.invoke(Method.java:597)
> >
> >           at
> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call
> > (WritableRpcEngine.java:474)
> >
> >           at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
> >
> >           at java.security.AccessController.doPrivileged(Native
> > Method)
> >
> >           at javax.security.auth.Subject.doAs(Subject.java:396)
> >
> >           at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> > ion.java:1332)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
> >
> >
> >
> > 2013-01-29 17:11:29,212 WARN org.apache.hadoop.security.token.Token:
> > Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
> >
> > 2013-01-29 17:11:29,212 ERROR
> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
> > Exception renewing tokenKind: MAPREDUCE_DELEGATION_TOKEN, Service:
> > 10.204.12.62:8021,
> > Ident: 00 04 68 64 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65
> > 8a 01 3c 88 94 58 67 8a 01 3c ac a0 dc 67 4c 08. Not rescheduled
> >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> > Client mapred tries to renew a token with renewer specified as mr
> > token
> >
> >           at
> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec
> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
> >
> >           at
> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja
> > va:3738)
> >
> >           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> >           at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:39)
> >
> >           at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:25)
> >
> >           at java.lang.reflect.Method.invoke(Method.java:597)
> >
> >           at
> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call
> > (WritableRpcEngine.java:474)
> >
> >           at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
> >
> >           at java.security.AccessController.doPrivileged(Native
> > Method)
> >
> >           at javax.security.auth.Subject.doAs(Subject.java:396)
> >
> >           at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> > ion.java:1332)
> >
> >           at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
> >
> >
> >
> >           at org.apache.hadoop.ipc.Client.call(Client.java:1160)
> >
> >           at
> > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
> > ne.java:225)
> >
> >           at
> > org.apache.hadoop.mapred.$Proxy12.renewDelegationToken(Unknown
> > Source)
> >
> >           at
> > org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:541)
> >
> >           at
> > org.apache.hadoop.security.token.Token.renew(Token.java:372)
> >
> >           at
> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene
> > walTimerTask$1.run(DelegationTokenRenewal.java:218)
> >
> >           at
> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene
> > walTimerTask$1.run(DelegationTokenRenewal.java:214)
> >
> >           at java.security.AccessController.doPrivileged(Native
> > Method)
> >
> >           at javax.security.auth.Subject.doAs(Subject.java:396)
> >
> >           at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> > ion.java:1332)
> >
> >           at
> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene
> > walTimerTask.run(DelegationTokenRenewal.java:213)
> >
> >           at java.util.TimerThread.mainLoop(Timer.java:512)
> >
> >           at java.util.TimerThread.run(Timer.java:462)
> >
> >
> >
> > ~Corbett Martin
> >
> > Software Architect
> >
> > AbsoluteAR Accounts Receivable Services - An NHIN Solution
> >
> >
> >
> >
> > ________________________________
> > This message and its contents (to include attachments) are the
> > property of National Health Systems, Inc. and may contain confidential
> > and proprietary information. This email and any files transmitted with
> > it are intended solely for the use of the individual or entity to whom
> they are addressed.
> > You are hereby notified that any unauthorized disclosure, copying, or
> > distribution of this message, or the taking of any unauthorized action
> > based on information contained herein is strictly prohibited.
> > Unauthorized use of information contained herein may subject you to
> > civil and criminal prosecution and penalties. If you are not the
> > intended recipient, you should delete this message immediately and
> > notify the sender immediately by telephone or by replying to this
> transmission.
>
>
>
> --
> Harsh J
>
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action
> based on information contained herein is strictly prohibited. Unauthorized
> use of information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you
> should delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
>



-- 
Alejandro

Mime
View raw message