hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Iyappan Srinivasan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
Date Fri, 30 Apr 2010 10:26:54 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862623#action_12862623
] 

Iyappan Srinivasan commented on MAPREDUCE-1741:
-----------------------------------------------

What is the reason we create manually 100 files, please provide the documentation, looking
at the code it is not evident to me. 

100 files are created for the test scenario in which the movemnt of history files to done
directory should still happen, even iftehre are 100 dummy files in that directory. 

In the multiple job scenario are the jobs launches simultaneously or sequentially one after
the other ?

What is the reason we create manually 100 files, please provide the documentation, looking
at the code it is not evident to me. 2. In the multiple job scenario are the jobs launches
simultaneously or sequentially one after the other ? 
- It is sequential, with a difference of few seconds. The test scenario is  when a bunch of
testcases finish almost at the same time, still ,the history files have to be moved to done
directory

Test names are bit confusing for me. Can you please change to some appropriate names? 
 Here is my suggestion.
testRetiredJobsCompletedLocation() - > testRetiredJobsHistoryLocation()
testRetiredMultipleJobsCompletedLocation() -> testRetiredMultipleJobsHistoryLocation()
- Done

String hadoopLogDirString = jconf.get("hadoop.log.dir");
jobHistoryDonePathString = jconf.get("mapred.job.tracker.history.completed.location");
String jobHistoryPathString = jconf.get("hadoop.job.history.location");

Can please check the above three statements with assert condition. Otherwise if any of these
attributes are not available in the configuration 
then test fails w/ NPE w/out any message. So please check the above three values before proceeding
to further steps

- hadoop.log.dir is not required . Removed it. 
- Added a Assert statement for mapred.job.tracker.history.completed.location
- hadoop.job.history.location is a optional parm , which may or may not be present. And it
si checked for null, before usage. So, no need for assert.

How do you confirm whether its moved to retired job state or not after completion of job ?
I don't see any specific condition in the code for retire job. Is there any default interval
available in the configuration for moving jobs to retire? 

- The retired jbo movement is done as a inline method and will happen soon after job is completed.
So, the code waits for a maximum of 30 seconds 
to make absolutely sure that it is moved to retired. 

Put the assert statement like below after completion of job.So that user will have clear information
whenever test fails.
Assert.assertTrue("Job history files are not available in history location after job retired.",
jobFileFound);

- It is checking in the done directory. Job history file should be found here. If it si not
found, then it asserts. I have elaborated the message as "Job History 
File is not found in the done directory"

Can you please put the annotation @Test after java doc comments instead of top.

[ Show ยป ] Vinay Kumar Thota added a comment - 30/Apr/10 02:06 AM 
Test names are bit confusing for me. Can you please change to some appropriate names? 
 Here is my suggestion.
testRetiredJobsCompletedLocation() - > testRetiredJobsHistoryLocation()
testRetiredMultipleJobsCompletedLocation() -> testRetiredMultipleJobsHistoryLocation()

String hadoopLogDirString = jconf.get("hadoop.log.dir");
jobHistoryDonePathString = jconf.get("mapred.job.tracker.history.completed.location");
String jobHistoryPathString = jconf.get("hadoop.job.history.location");

Can please check the above three statements with assert condition. Otherwise if any of these
attributes are not available in the configuration 
then test fails w/ NPE w/out any message. So please check the above three values before proceeding
to further steps

Assert.assertTrue("jobFileFound is false", jobFileFound);

You are checking the above condition while running the job right, in this case the above statement
would be wrong because 
the jobFileFound should be false and it should be like below.

Assert.assertFalse("Job history files available in history location for running job itself.",jobFileFound)

How do you confirm whether its moved to retired job state or not after completion of job ?
I don't see any specific condition in the code for retire job. Is there any default interval
available in the configuration for moving jobs to retire? 
Put the assert statement like below after completion of job.So that user will have clear information
whenever test fails.
Assert.assertTrue("Job history files are not available in history location after job retired.",
jobFileFound);

Even I have same question that Balaji has asked regarding creating 1000 files in done location.
Can you please elaborate more on this in your comments. Can you please put the annotation
@Test after java doc comments instead of top. 

- Done







> Automate the test scenario of  job related files are moved from history directory to
done directory
> ---------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1741
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741
>             Project: Hadoop Map/Reduce
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>             Fix For: 0.22.0
>
>         Attachments: TestJobHistoryLocation.patch
>
>
> Job related files are moved from history directory to done directory, when
> 1) Job succeeds
> 2) Job is killed
> 3) When 100 files are put in the done directory
> 4) When multiple jobs are completed at the same time, some successful, some failed.
> Also, two files, conf.xml and job files should be present in the done directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message