hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuanbo Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-7220) Mapreduce jobhistory summary error if job name is very long
Date Mon, 17 Jun 2019 03:36:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yuanbo Liu updated MAPREDUCE-7220:
----------------------------------
    Issue Type: Bug  (was: Improvement)

> Mapreduce jobhistory summary error if job name is very long
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-7220
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7220
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Yuanbo Liu
>            Priority: Critical
>
> From JobHistoryEventHandler.java, we can see that mapreduce uses writeUTF to write summary.done
file to hdfs. The code is here:
> {quote}summaryFileOut = doneDirFS.create(qualifiedSummaryDoneFile, true);
> summaryFileOut.writeUTF(mi.getJobSummary().getJobSummaryString());
> summaryFileOut.close();
> {quote}
> writeUTF uses first two bytes to record string length, hence the length of summary string
cannot exceed 65535. But in the case of hive job, SQL string is part of job name. It's quite
normal that SQL length is greater than 65535, then summary done file cannot be written successfully.
In this case, hive client thinks such kind of mapreduce job is in the final state of failure
sometimes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message