hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar Vadali (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-1819) RaidNode should be smarter in submitting Raid jobs
Date Sat, 02 Oct 2010 05:15:32 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ramkumar Vadali updated MAPREDUCE-1819:
---------------------------------------

    Attachment: MAPREDUCE-1819.4.patch

Return true directly in shouldSelectFiles(), merge changes with recent commits after svn up.

Test Results:
===========

ant test-patch


1819 latest


     [exec] 
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 16 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of
release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
     [exec] 
     [exec] 
     [exec] 
     [exec] 
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================

ant test:

There were test failures, but those failures were present in a clean checkout as well. The
patch does not introduce any new failures. The following are the tests that failed on a clean
checkout:

    [junit] Test org.apache.hadoop.mapred.TestMapRed FAILED
    [junit] Test org.apache.hadoop.mapred.TestSequenceFileInputFormat FAILED
    [junit] Test org.apache.hadoop.io.TestSequenceFile FAILED
    [junit] Test org.apache.hadoop.mapred.TestControlledMapReduceJob FAILED (timeout)
    [junit] Test org.apache.hadoop.mapred.TestJobQueueInformation FAILED
    [junit] Test org.apache.hadoop.mapred.TestSequenceFileAsTextInputFormat FAILED
    [junit] Test org.apache.hadoop.mapred.TestSequenceFileInputFilter FAILED
    [junit] Test org.apache.hadoop.mapreduce.TestMapReduce FAILED
    [junit] Test org.apache.hadoop.mapreduce.lib.input.TestMRSequenceFileAsTextInputFormat
FAILED
    [junit] Test org.apache.hadoop.mapreduce.lib.input.TestMRSequenceFileInputFilter FAILED
    [junit] Test org.apache.hadoop.record.TestRecordMR FAILED


> RaidNode should be smarter in submitting Raid jobs
> --------------------------------------------------
>
>                 Key: MAPREDUCE-1819
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1819
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: contrib/raid
>    Affects Versions: 0.20.1
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-1819.4.patch, MAPREDUCE-1819.patch, MAPREDUCE-1819.patch.2,
MAPREDUCE-1819.patch.3
>
>
> The RaidNode currently computes parity files as follows:
> 1. Using RaidNode.selectFiles() to figure out what files to raid for a policy
> 2. Using #1 repeatedly for each configured policy to accumulate a list of files. 
> 3. Submitting a mapreduce job with the list of files from #2 using DistRaid.doDistRaid()
> This task addresses the fact that #2 and #3 happen sequentially. The proposal is to submit
a separate mapreduce job for the list of files for each policy and use another thread to track
the progress of the submitted jobs. This will help reduce the time taken for files to be raided.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message