hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4191) Add a testcase for jobhistory
Date Wed, 04 Feb 2009 20:15:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670443#action_12670443
] 

Ravi Gummadi commented on HADOOP-4191:
--------------------------------------

Here is a proposal for the new testcase for job history:

Run a job and then parse the job history file and validate the following:
(1) history file name and location are validated.
(2) Each line's format is checked against a regex.
(3) Format of all TIMEs are checked against a regex.
(4) Checking if the VALUE in history file for a partivcular KEY is valid, specifically for
KEYs like JOB_STATUS, TASK_TYPE, TASK_STATUS, JOB_PRIORITY.
(5) Check if changes to properties like Job_PRIORITY are logged to the history file.
(6) We maintain a Map<String, List<String>> of TaskIDs to TaskAttemptIDs and see
if all the atemptIDs are finished before the TaskID finishes.
        We match finish of each TaskAttemptIDs, TaskIDs against its start to make sure that
all TaskAttempts, Tasks and Jobs started are
        indeed finished and the history log lines for them are in proper order.


Thoughts ?

> Add a testcase for jobhistory
> -----------------------------
>
>                 Key: HADOOP-4191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4191
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred, test
>            Reporter: Amar Kamat
>            Assignee: Ravi Gummadi
>
> Changes in job history might break the history parser which in turn might break some
features like jobtracker-recovery, history-viewer etc. There should be a testcase that catches
these incompatible changes early and informs about the expected change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message