hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6950) Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
Date Fri, 15 Dec 2017 20:39:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Konstantin Shvachko updated MAPREDUCE-6950:
-------------------------------------------
    Fix Version/s:     (was: 2.7.5)

> Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6950
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6950
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am
>    Affects Versions: 2.7.1
>            Reporter: zhengchenyu
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> some job report error, like this:
> {code}
> hadoop.mapreduce.Job.monitorAndPrintJob(Job.java 1367) [main] :  map 100% reduce 100%
> [2017-08-31T20:27:12.591+08:00] [INFO] hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java
277) [main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting
to job history server
> [2017-08-31T20:27:12.821+08:00] [INFO] hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java
277) [main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting
to job history server
> [2017-08-31T20:27:13.039+08:00] [INFO] hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java
277) [main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting
to job history server
> [2017-08-31T20:27:13.256+08:00] [ERROR] hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java
1034) [main] : Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
> {code}
> I found the am container log, like below. Here we know error happened in pipeline, maybe
some dn error. And I also found some other reason which close the JobHistoryEventHandler.
So MR AM can't write the information for JH. So client counldn't know whether the appplication
is finished. 
> {code}
> 2017-08-31 20:27:10,813 INFO [Thread-1968] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event MAP_ATTEMPT_STARTED
> 2017-08-31 20:27:10,814 ERROR [Thread-1968] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Error writing History Event: org.apache.hadoop.mapreduce.jobhistory.TaskAttemptStartedEvent@2055ea0a
> java.io.EOFException: Premature EOF: no length prefix available
>         at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2292)
>         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1317)
>         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)
>         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> 2017-08-31 20:27:10,814 INFO [Thread-1968] org.apache.hadoop.service.AbstractService:
Service JobHistoryEventHandler failed in state STOPPED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.EOFException: Premature EOF: no length prefix available
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.EOFException: Premature
EOF: no length prefix available
>         at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:580)
>         at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:374)

>         at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
>         at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
>         at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
>         at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
>         at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> {code}
> This problem is serious , especially for hive. Job must rerun meaninglessly!  So I think
we need to retry the operation of writing history event. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message