hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chin-Jung Hsu <oxhead.l...@gmail.com>
Subject Re: Maven build YARN ResourceManager only
Date Sat, 13 Apr 2013 14:10:41 GMT
Hi Chris,

Appreciate your help, and sorry for the crossposting on both 'user' and
'yarn-dev'.  I first posted on yarn-dev and didn't see anything.  I then
thought I might not be able to post on that list.  That's why I posted it
again on 'user'.  Should I post this kind of questions on 'yarn-dev' or
'user'?

Right now, I cannot compile hadoop-yarn-server-resourcemanager independently.
 I have to use

mvn package -Pdist -DskipTests -Dtar

to compile the whole project.  What command is appropriate to this scenario?


Thanks,
oxhead




On Sat, Apr 13, 2013 at 12:41 AM, Chris Nauroth <cnauroth@hortonworks.com>wrote:

> I don't have an answer to your exact question, but I do have a different
> suggestion that prevents the need to do frequent rebuilds of the whole
> Hadoop source tree.  First, do a full build of the distribution tar.gz.
>  Extract it and set up a custom hadoop-env.sh for yourself.  Inside the
> hadoop-env.sh file, export the environment variables
> HADOOP_USER_CLASSPATH_FIRST=true and HADOOP_CLASSPATH= any classpath that
> you want to prepend before the classes loaded from the distribution.  For
> example, this is what I have in mine right now, because I'm mostly working
> on HDFS and NodeManager:
>
> export HADOOP_USER_CLASSPATH_FIRST=true
> HADOOP_REPO=~/git/hadoop-common
> export
> HADOOP_CLASSPATH=$HADOOP_REPO/hadoop-common-project/hadoop-common/target/classes:$HADOOP_REPO/hadoop-hdfs-project/hadoop-hdfs/target/classes:$HADOOP_REPO/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/classes:$HADOOP_REPO/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
>
> For your ResourceManager work, you could set up your HADOOP_CLASSPATH to
> point at your hadoop-yarn-server-resourcemanager/target/classes directory.
>  Then, source (.) this hadoop-env.sh in any shell that you're using to run
> hadoop commands.  The daemons will print their full classpath before
> launching, so you can check that to see if it worked.
>
> With all of this in place, you can keep recompiling just
> hadoop-yarn-server-resourcemanager whenever you make changes instead of the
> whole hadoop-common tree.  Does this help?
>
> Thanks,
> --Chris
>
>
> On Fri, Apr 12, 2013 at 8:46 PM, Chin-Jung Hsu <oxhead.list@gmail.com>wrote:
>
>> I am implementing my own YARN scheduler under 2.0.3-alpha.  Is that
>> possible to build only the ResourceManager project, and then create a
>> distribution tar.gz for the entire Hadoop project?  Right now, the
>> compiling time takes me about 9 minutes.
>>
>> Thanks,
>> oxhead
>>
>
>

Mime
View raw message