hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar Vadali (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1819) RaidNode should be smarter in submitting Raid jobs
Date Thu, 07 Oct 2010 22:55:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919092#action_12919092
] 

Ramkumar Vadali commented on MAPREDUCE-1819:
--------------------------------------------

I re-ran ant test-patch, that succeeded. I ran ant test-patch under src/contrib/raid and that
succeeded too. Since the patch touches code under src/contrib/raid only, I did not run ant
test from the top level. I think that should be OK.

     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 16 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of
release audit warnings.
     [exec]
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
     [exec]
     [exec]
     [exec]
     [exec]
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]
     [exec]

BUILD SUCCESSFUL
Total time: 18 minutes 17 seconds


test-junit:
    [junit] WARNING: multiple versions of ant detected in path for junit 
    [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
    [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
    [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 37.74 sec
    [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 8.185 sec
    [junit] Running org.apache.hadoop.raid.TestRaidHar
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 68.763 sec
    [junit] Running org.apache.hadoop.raid.TestRaidNode
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 464.922 sec
    [junit] Running org.apache.hadoop.raid.TestRaidPurge
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 44.036 sec

test:

BUILD SUCCESSFUL
Total time: 10 minutes 38 seconds

> RaidNode should be smarter in submitting Raid jobs
> --------------------------------------------------
>
>                 Key: MAPREDUCE-1819
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1819
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: contrib/raid
>    Affects Versions: 0.20.1
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-1819.4.patch, MAPREDUCE-1819.patch, MAPREDUCE-1819.patch.2,
MAPREDUCE-1819.patch.3
>
>
> The RaidNode currently computes parity files as follows:
> 1. Using RaidNode.selectFiles() to figure out what files to raid for a policy
> 2. Using #1 repeatedly for each configured policy to accumulate a list of files. 
> 3. Submitting a mapreduce job with the list of files from #2 using DistRaid.doDistRaid()
> This task addresses the fact that #2 and #3 happen sequentially. The proposal is to submit
a separate mapreduce job for the list of files for each policy and use another thread to track
the progress of the submitted jobs. This will help reduce the time taken for files to be raided.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message