hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7737) normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs
Date Thu, 13 Oct 2011 20:17:11 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126880#comment-13126880
] 

Tom White commented on HADOOP-7737:
-----------------------------------

+1 I tried with and without -Dtar. I managed to run a MR job from the "exploded" directory.

{noformat}
export HADOOP_COMMON_HOME=$(pwd)/$(ls -d hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
export HADOOP_HDFS_HOME=$(pwd)/$(ls -d hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
export HADOOP_MAPRED_HOME=$(pwd)/$(ls -d hadoop-mapreduce-project/target/hadoop-mapreduce-*-SNAPSHOT)
export YARN_HOME=$HADOOP_MAPRED_HOME

export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$HADOOP_MAPRED_HOME/bin:$PATH

cat > $YARN_HOME/conf/yarn-site.xml  << EOF
<?xml version="1.0"?>
<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce.shuffle</value>
    </property>
    <property>
      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>
EOF

cd hadoop-mapreduce-project
ant examples -Dresolvers=internal
cd ..
export HADOOP_CLASSPATH=$YARN_HOME/modules/*
mkdir in
cp BUILDING.txt in/
hadoop jar hadoop-mapreduce-project/build/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar wordcount
-Dmapreduce.job.user.name=$USER in out
{noformat}

I'll add this to http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment after
it's committed.
                
> normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-7737
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7737
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0, 0.24.0
>
>         Attachments: HADOOP-7737.patch, HADOOP-7737.patch
>
>
> Normalize the build fo hadoop-mapreduce and hadoop-dist with hadoop-common and hadoop-hdfs
making the -Pdist and -Dtar maven options to be consistent.
> * -Pdist should create the layout
> * -Dtar should create the TAR

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message