hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3868) Reenable Raid
Date Thu, 16 Feb 2012 19:03:07 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209614#comment-13209614

Alejandro Abdelnur commented on MAPREDUCE-3868:


A few tips that may be useful when doing the integration:

* If using and IDE (I use Idea) you may have to change the scope of hadoop-minicluster to
compile for the IDE to pick all necessary JARs (Idea seems to get confused otherwise). From
the command line (Maven) things work fine with the (correct) test scope.
* You don't need to build the whole Hadoop project every time. Instead do the following:
** Run once, from root level 'mvn install -DskipTests'
** then CD to hadoop-hdfs-project/hadoop-hdsf-raid and work from there.
** You can do Maven command or point your IDE to the POM there.
** Make sure you are using -offline with Maven command and in your IDE (else you may pick
up newer snapshot JARs/POMs from Apache that don't have the changes this patch added)
* The new YARN base MiniCluster still has some issues, on failures you have  runaway processes
(the shutdown logic is broken, a JIRA to fix that is in the works MAPREDUCE-3634)
** What i do is a 'ps -ef | grep java' and then kill the dangling processes
* Sometimes (at least in Mac OS) MiniDFS finds some local files and things it is locked and
testcases fail because of that. If that is the case, delete those files.
* After doing a build javadocs stuff sometimes creates a dir api at src level, ignore that
DIR (don't add it to your patch)
* the distribution layout follows the new layout, I've put the scripts in the right places,
but I have not modified the scripts to look for things in the right places
* the full distribution must be done from root level, do 'mvn clean package -DskipTests -Pdist',
the full distribution will be in hadoop-dist/target/....
* the testcases that start MR jobs (and maybe the main code) will have to setup the framework
to YARN, you have to set in your job conf the following property: mapreduce.framework.name=yarn
(the streaming porting did this HADOOP-7590)
* in the testcases, you may have to add your classes as attached JAR, use the JarFinder.getJar()
class to get the RAID JAR (the streaming porting did this HADOOP-7590).
* Some MiniMRCluster methods related to MR1 throw UnsupportedOperationExceptions, you may
have to find other ways of getting the right info. For example: MiniMRCluster.getJobTrackerPort()
now throws UOE, you should use instead MiniMRCluster.createJobConf().get(JTConfig.JT_IPC_ADDRESS)
* I've modified the TestBlockFixer with the last 2 items and TestBlockFixerBlockFixDist.testBlockFixDist()
test get to start and run for long before failing.
> Reenable Raid
> -------------
>                 Key: MAPREDUCE-3868
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/raid
>            Reporter: Scott Chen
>            Assignee: Ramkumar Vadali
>         Attachments: MAPREDUCE-3868v1.patch, MAPREDUCE-3868v1.sh
> Currently Raid is outdated and not compiled. Make it compile.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message