hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhengchenyu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6950) Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
Date Fri, 01 Sep 2017 03:55:02 GMT
zhengchenyu created MAPREDUCE-6950:
--------------------------------------

             Summary: Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
                 Key: MAPREDUCE-6950
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6950
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mr-am
    Affects Versions: 2.7.1
            Reporter: zhengchenyu
             Fix For: 2.7.5


some job report error, like this:
{code}
hadoop.mapreduce.Job.monitorAndPrintJob(Job.java 1367) [main] :  map 100% reduce 100%
[2017-08-31T20:27:12.591+08:00] [INFO] hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java
277) [main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting
to job history server
[2017-08-31T20:27:12.821+08:00] [INFO] hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java
277) [main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting
to job history server
[2017-08-31T20:27:13.039+08:00] [INFO] hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java
277) [main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting
to job history server
[2017-08-31T20:27:13.256+08:00] [ERROR] hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java
1034) [main] : Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
{code}

I found the am container log, like below. Here we know error happened in pipeline, maybe some
dn error. And I also found some other reason which close the JobHistoryEventHandler. So MR
AM can't write the information for JH. So client counldn't know whether the appplication is
finished. 

{code}
2017-08-31 20:27:10,813 INFO [Thread-1968] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event MAP_ATTEMPT_STARTED
2017-08-31 20:27:10,814 ERROR [Thread-1968] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Error writing History Event: org.apache.hadoop.mapreduce.jobhistory.TaskAttemptStartedEvent@2055ea0a
java.io.EOFException: Premature EOF: no length prefix available
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2292)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1317)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
2017-08-31 20:27:10,814 INFO [Thread-1968] org.apache.hadoop.service.AbstractService: Service
JobHistoryEventHandler failed in state STOPPED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.EOFException: Premature EOF: no length prefix available
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.EOFException: Premature EOF:
no length prefix available
        at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:580)
        at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:374)

        at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
        at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
        at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
        at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
{code}

This problem is serious , especially for hive. Job must rerun meaninglessly!  So I think we
need to retry the operation of writing history event. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message