hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4191) Add a testcase for jobhistory
Date Wed, 11 Feb 2009 04:35:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672502#action_12672502
] 

Amareshwari Sriramadasu commented on HADOOP-4191:
-------------------------------------------------

some comments:
1. Any reason for using LocalFileSystem for the tests?
bq.       mr = new MiniMRCluster(2, "file:///", 3);
I thought the tests should be done with DFS.

2. should we have a test with history location on dfs?

3. I think the test is still missing validation for some fieds such as SPLITS, because earlier
we have seen bugs where history stopped logging splits.

4. I propose we should validate all the fields of atleast one map task and one reduce task.
For map task attempt, we should validate  TASK_ATTEMPT_ID, TASKID, START_TIME, FINISH_TIME,
TRACKER_NAME, HTTP_PORT, TASK_STATUS, HOSTNAME, STATE_STRING, COUNTERS, ERROR (in case of
failure)
For map task, we should validate TASKID, START_TIME, FINISH_TIME, TASK_STATUS, SPLITS, COUNTERS.
For reduce task attempt, we should validate TASK_ATTEMPT_ID, TASKID, START_TIME, SHUFFLE_FINISHED,
SORT_FINISHED, FINISH_TIME, TRACKER_NAME, HTTP_PORT, TASK_STATUS, HOSTNAME, STATE_STRING,
COUNTERS, ERROR (in case of failure).

Since RunningJob handle and JobClient (you can create one) handle are available, we could
get the expected values for all these and validate the values in history.
Thoughts?

5. If the test case is looking too big, we can refactor the tests into different files. thoughts?

> Add a testcase for jobhistory
> -----------------------------
>
>                 Key: HADOOP-4191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4191
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred, test
>            Reporter: Amar Kamat
>            Assignee: Ravi Gummadi
>         Attachments: HADOOP-4191.patch, HADOOP-4191.v2.patch
>
>
> Changes in job history might break the history parser which in turn might break some
features like jobtracker-recovery, history-viewer etc. There should be a testcase that catches
these incompatible changes early and informs about the expected change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message