hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rohit sarewar <rohitsare...@gmail.com>
Subject Redirecting to job history server fails in MapReduceV2
Date Wed, 23 Jul 2014 08:39:11 GMT
Hi

 I am facing an error during Java Map reduce code execution in using Hadoop
2.2
The FinalApplicationStatus is successful. I can find the output in desired
location but It throws error while redirecting to job history server
(please refer to red line in the logs below).

*Explanation in detail:*
Input File to the Java MR job:
100 files ~ each with 300 MB size , Total size = 34 GB

*Case 1: (1 Job ) Error*
Run one MR job  for all 100 files as input to the job ( Run time - 1 hour)
It throws error as given in job logs - marked in red color below.

*Case 2: (100 job)Successful*
Run one MR job  for each 100 files as input to the job ( Run time - 40 sec
per job)
I can see all 100 output and all job ran successfully.Also for all these
jobs it successfully redirected to job history server and displayed all
counter in terminal. (No error was seen in this case)

After Case 2, I cannot blame to my cluster configuration.

*How do I fix this ? Please help.*

I could also find few unresolved JIRA tickets and discussion for similar
case(links below):
1) Similar case
http://mail-archives.apache.org/mod_mbox/oozie-user/201402.mbox/%3CCAAu13zFQWQuVo-ShJYVT-o+D=SC6-4Zhwn8UKPsB31wS62=0Jw@mail.gmail.com%3E

2) Exact Same Case
http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201311.mbox/%3CCANXCz3RC5aCgeAMro=wS6TbwjsQVx9kemJd28kpmjMTp5hdX=Q@mail.gmail.com%3E

3) Similar Case
http://stackoverflow.com/questions/17930644/connection-error-in-apache-pig


*Job Logs:*

14/07/22 06:37:11 INFO mapreduce.Job:  map 100% reduce 98%

14/07/22 06:37:44 INFO mapreduce.Job:  map 100% reduce 99%

14/07/22 06:38:30 INFO mapreduce.Job:  map 100% reduce 100%

14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server

14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server

14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server

14/07/22 06:39:02 ERROR security.UserGroupInformation:
PriviledgedActionException as: rohitsarewar (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException

        at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269)

        at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)

        at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)

        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)



Exception in thread "main" java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException

        at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269)

        at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)

        at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)

        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)



        at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:330)

        at
org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:382)

        at
org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:529)

        at org.apache.hadoop.mapreduce.Job$5.run(Job.java:668)

        at org.apache.hadoop.mapreduce.Job$5.run(Job.java:665)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

        at
org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:665)

        at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1349)

        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)

        at com.bigdata.mapreduce.esc.escDriver.main(escDriver.java:23)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): j

        at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolH

        at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBSer

        at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.

        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(P

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)



        at org.apache.hadoop.ipc.Client.call(Client.java:1347)

        at org.apache.hadoop.ipc.Client.call(Client.java:1300)

        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine

        at com.sun.proxy.$Proxy12.getTaskAttemptCompletionEvents(Unknown
Source)

        at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClie

        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDeleg

        ... 16 more


Regards

Rohit Sarewar

Mime
View raw message