hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2403) JobHistory log files contain data that cannot be parsed by org.apache.hadoop.mapred.JobHistory
Date Tue, 12 Aug 2008 08:44:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621734#action_12621734

Amar Kamat commented on HADOOP-2403:

I think we should fix the general problem to do with history parsing which are
1) Detect if the record is complete or not. The client can fail while writing to the history
and the failure can be exactly on the key-val boundary.
2) Detect if the key-val pairs are correct. The error message can contain tabs and other characters
like {{"}} which can error the history parsing. Currently  a tab is the delimiter for records
and a {{"}} is used for value encapsulation. Similarly other strings in the history can have
these characters like counter-names etc. 
This problems can fail HADOOP-3245.

I would go for having a record _delimiter_ like a {{.}}(dot) to detect if the record is complete
or not. Also incomplete records should not be parsed and should be ignored. We also need to
make sure that the characters that are used as delimiter ({{.}}, {{"}}, tab) should not occur
in a _value_.

> JobHistory log files contain data that cannot be parsed by org.apache.hadoop.mapred.JobHistory
> ----------------------------------------------------------------------------------------------
>                 Key: HADOOP-2403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2403
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Runping Qi
>            Assignee: Amareshwari Sriramadasu
>            Priority: Critical
>             Fix For: 0.19.0
>         Attachments: EncodeDecode.java, patch-2403.txt
> When some tasks failed, the job tracker writes an line to the history file with error
> However, the error message may mess up with the history file format, choking the history
parser. Here is an example:
> MapAttempt TASK_TYPE="MAP" TASKID="tip_200712102254_0001_m_000090" TASK_ATTEMPT_ID="task_200712102254_0001_m_000090_0"
TASK_STATUS="FAILED" FINISH_TIME="1197327293253" HOSTNAME="XXXX:50050" ERROR="java.lang.IllegalArgumentException:
Trouble to get key or value (<,> substituted by null 
> . Key XML-Ori:
>         <Root>

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message