Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 692F5FA42 for ; Sat, 13 Apr 2013 14:11:17 +0000 (UTC) Received: (qmail 51564 invoked by uid 500); 13 Apr 2013 14:11:08 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 51408 invoked by uid 500); 13 Apr 2013 14:11:08 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 51244 invoked by uid 99); 13 Apr 2013 14:11:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Apr 2013 14:11:08 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of oxhead.list@gmail.com designates 209.85.210.175 as permitted sender) Received: from [209.85.210.175] (HELO mail-ia0-f175.google.com) (209.85.210.175) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Apr 2013 14:11:02 +0000 Received: by mail-ia0-f175.google.com with SMTP id e16so3121051iaa.20 for ; Sat, 13 Apr 2013 07:10:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=ODmcY7mF4dyYoOCSdUmYCRGMPu6sAD9yghyNo5oqq/8=; b=EhZ9uDdO8cigE9aoU4cBdf8jWiDnwBO+zAHHV5OtmAhyG9ml6ZjwpAA7X4m58cfN4D Ox1e3LYXNu7Cyf/FYAE02Deqz6c8fie2uUrXYF5DUKT/ex83aXIuhawelw69DuqK6Q5r SEui6us8dX32wDzP1br/ZQxZmOjz3wRlxiCNaVNpCGp1y9D+o/IB4/MOXklmZ/Dm6gcR 5a79C70ak9UqAUTm6pU1EWe2IonTjSOmJ0LBEOK9r8bc5JsocLpaOAr0dTzkpt3cGsce r7dWbh78ghpZitxND9X4KZRw3uC2RCIP4QIv3wl0NPbDlujXw51JoruD+31hBa/KkwDu s/Cg== MIME-Version: 1.0 X-Received: by 10.50.36.97 with SMTP id p1mr1638314igj.64.1365862241299; Sat, 13 Apr 2013 07:10:41 -0700 (PDT) Received: by 10.64.102.202 with HTTP; Sat, 13 Apr 2013 07:10:41 -0700 (PDT) In-Reply-To: References: Date: Sat, 13 Apr 2013 10:10:41 -0400 Message-ID: Subject: Re: Maven build YARN ResourceManager only From: Chin-Jung Hsu To: user@hadoop.apache.org Cc: yarn-dev@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e011609a4b4735704da3e953f X-Virus-Checked: Checked by ClamAV on apache.org --089e011609a4b4735704da3e953f Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Chris, Appreciate your help, and sorry for the crossposting on both 'user' and 'yarn-dev'. I first posted on yarn-dev and didn't see anything. I then thought I might not be able to post on that list. That's why I posted it again on 'user'. Should I post this kind of questions on 'yarn-dev' or 'user'? Right now, I cannot compile hadoop-yarn-server-resourcemanager independentl= y. I have to use mvn package -Pdist -DskipTests -Dtar to compile the whole project. What command is appropriate to this scenario= ? Thanks, oxhead On Sat, Apr 13, 2013 at 12:41 AM, Chris Nauroth w= rote: > I don't have an answer to your exact question, but I do have a different > suggestion that prevents the need to do frequent rebuilds of the whole > Hadoop source tree. First, do a full build of the distribution tar.gz. > Extract it and set up a custom hadoop-env.sh for yourself. Inside the > hadoop-env.sh file, export the environment variables > HADOOP_USER_CLASSPATH_FIRST=3Dtrue and HADOOP_CLASSPATH=3D any classpath = that > you want to prepend before the classes loaded from the distribution. For > example, this is what I have in mine right now, because I'm mostly workin= g > on HDFS and NodeManager: > > export HADOOP_USER_CLASSPATH_FIRST=3Dtrue > HADOOP_REPO=3D~/git/hadoop-common > export > HADOOP_CLASSPATH=3D$HADOOP_REPO/hadoop-common-project/hadoop-common/targe= t/classes:$HADOOP_REPO/hadoop-hdfs-project/hadoop-hdfs/target/classes:$HADO= OP_REPO/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/classes:$= HADOOP_REPO/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-= server-nodemanager/target/classes > > For your ResourceManager work, you could set up your HADOOP_CLASSPATH to > point at your hadoop-yarn-server-resourcemanager/target/classes directory= . > Then, source (.) this hadoop-env.sh in any shell that you're using to ru= n > hadoop commands. The daemons will print their full classpath before > launching, so you can check that to see if it worked. > > With all of this in place, you can keep recompiling just > hadoop-yarn-server-resourcemanager whenever you make changes instead of t= he > whole hadoop-common tree. Does this help? > > Thanks, > --Chris > > > On Fri, Apr 12, 2013 at 8:46 PM, Chin-Jung Hsu wro= te: > >> I am implementing my own YARN scheduler under 2.0.3-alpha. Is that >> possible to build only the ResourceManager project, and then create a >> distribution tar.gz for the entire Hadoop project? Right now, the >> compiling time takes me about 9 minutes. >> >> Thanks, >> oxhead >> > > --089e011609a4b4735704da3e953f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Chris,

Appreciate your help, and sor= ry for the crossposting on both 'user' and 'yarn-dev'. =A0I= first posted on yarn-dev and didn't see anything. =A0I then thought I = might not be able to post on that list. =A0That's why I posted it again= on 'user'. =A0Should I post this kind of questions on 'yarn-de= v' or 'user'?

Right now, I cannot compile=A0hadoop-yarn-server= -resourcemanager=A0independently. =A0I have to use=A0

mvn package -Pdist = -DskipTests -Dtar

to compile the whol= e project. =A0What command is appropriate to this scenario?


<= /div>
Thanks,
oxhead




On Sat, Apr 13, 2013 at 12:41 AM, Chris Nauroth <cnauroth@hortonworks.com> wrote:
I don't have an answer to your exact que= stion, but I do have a different suggestion that prevents the need to do fr= equent rebuilds of the whole Hadoop source tree. =A0First, do a full build = of the distribution tar.gz. =A0Extract it and set up a custom hadoop-env.sh= for yourself. =A0Inside the hadoop-env.sh file, export the environment var= iables HADOOP_USER_CLASSPATH_FIRST=3Dtrue and HADOOP_CLASSPATH=3D any class= path that you want to prepend before the classes loaded from the distributi= on. =A0For example, this is what I have in mine right now, because I'm = mostly working on HDFS and NodeManager:

export HADOOP_USER_CLASSPATH_FIRST=3Dtrue
HAD= OOP_REPO=3D~/git/hadoop-common
export HADOOP_CLASSPATH=3D$HADOOP_= REPO/hadoop-common-project/hadoop-common/target/classes:$HADOOP_REPO/hadoop= -hdfs-project/hadoop-hdfs/target/classes:$HADOOP_REPO/hadoop-yarn-project/h= adoop-yarn/hadoop-yarn-common/target/classes:$HADOOP_REPO/hadoop-yarn-proje= ct/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/cla= sses

For your ResourceManager work, you could set up your HA= DOOP_CLASSPATH to point at your hadoop-yarn-server-resourcemanager/target/c= lasses directory. =A0Then, source (.) this hadoop-env.sh in any shell that = you're using to run hadoop commands. =A0The daemons will print their fu= ll classpath before launching, so you can check that to see if it worked.

With all of this in place, you can keep recompiling jus= t hadoop-yarn-server-resourcemanager whenever you make changes instead of t= he whole hadoop-common tree. =A0Does this help?

Thanks,
--Chris


On Fri, Apr 12, 2013 at 8:46 PM, Chin-Jung Hsu <oxhead.list@gmail.com> wrote:
I am implementin= g my own YARN scheduler under 2.0.3-alpha. =A0Is that possible to build onl= y the ResourceManager project, and then create a distribution tar.gz for th= e entire Hadoop project? =A0Right now, the compiling time takes me about 9 = minutes.=A0

Thanks,
oxhead


--089e011609a4b4735704da3e953f--