Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9C4D21042F for ; Fri, 31 May 2013 22:19:30 +0000 (UTC) Received: (qmail 77620 invoked by uid 500); 31 May 2013 22:19:25 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 77293 invoked by uid 500); 31 May 2013 22:19:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77285 invoked by uid 99); 31 May 2013 22:19:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 May 2013 22:19:25 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=HTML_MESSAGE,PLING_QUERY,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of john.lilley@redpoint.net designates 206.225.164.223 as permitted sender) Received: from [206.225.164.223] (HELO hub021-nj-7.exch021.serverdata.net) (206.225.164.223) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 May 2013 22:19:19 +0000 Received: from MBX021-E3-NJ-2.exch021.domain.local ([10.240.4.78]) by HUB021-NJ-7.exch021.domain.local ([10.240.4.114]) with mapi id 14.03.0123.003; Fri, 31 May 2013 15:18:57 -0700 From: John Lilley To: "user@hadoop.apache.org" Subject: RE: built hadoop! please help with next steps? Thread-Topic: built hadoop! please help with next steps? Thread-Index: Ac5dUzyF6bTKBcl+R+WgLpfA87u/hgBEd1oAAAZpEWA= Date: Fri, 31 May 2013 22:18:57 +0000 Message-ID: <869970D71E26D7498BDAC4E1CA92226B658B75D3@MBX021-E3-NJ-2.exch021.domain.local> References: <869970D71E26D7498BDAC4E1CA92226B658B5060@MBX021-E3-NJ-2.exch021.domain.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [173.160.43.61] Content-Type: multipart/alternative; boundary="_000_869970D71E26D7498BDAC4E1CA92226B658B75D3MBX021E3NJ2exch_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_869970D71E26D7498BDAC4E1CA92226B658B75D3MBX021E3NJ2exch_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Sandy, Thanks for all of the tips, I will try this over the weekend. Regarding t= he last question, I am still trying to get the source loaded into Eclipse i= n a manner that facilitates easier browsing, symbol search, editing, etc. = Perhaps I am just missing some obvious FAQ? This is leading up to modifyin= g and debugging the "shell" ApplicationMaster sample. This page: http://stackoverflow.com/questions/11007423/developing-testing-and-debuggin= g-hadoop-map-reduce-jobs-with-eclipse looks promising as a Hadoop-in-Eclipse strategy, but it is over a year old = and I'm not sure if it applies to Hadoop 2.0 and YARN. John From: Sandy Ryza [mailto:sandy.ryza@cloudera.com] Sent: Friday, May 31, 2013 12:13 PM To: user@hadoop.apache.org Subject: Re: built hadoop! please help with next steps? Hi John, Here's how I deploy/debug Hadoop locally: To build and tar Hadoop: mvn clean package -Pdist -Dtar -DskipTests=3Dtrue The tar will be located in the project directory under hadoop-dist/target/.= I untar it into my deploy directory. I then copy these scripts into the same directory: hadoop-dev-env.sh: --- #!/bin/bash export HADOOP_DEV_HOME=3D`pwd` export HADOOP_MAPRED_HOME=3D${HADOOP_DEV_HOME} export HADOOP_COMMON_HOME=3D${HADOOP_DEV_HOME} export HADOOP_HDFS_HOME=3D${HADOOP_DEV_HOME} export YARN_HOME=3D${HADOOP_DEV_HOME} export HADOOP_CONF_DIR=3D${HADOOP_DEV_HOME}/etc/hadoop hadoop-dev-setup.sh: --- #!/bin/bash source ./hadoop-dev-env.sh bin/hadoop namenode -format hadoop-dev.sh: --- source hadoop-dev-env.sh sbin/hadoop-daemon.sh $1 namenode sbin/hadoop-daemon.sh $1 datanode sbin/yarn-daemon.sh $1 resourcemanager sbin/yarn-daemon.sh $1 nodemanager sbin/mr-jobhistory-daemon.sh $1 historyserver sbin/httpfs.sh $1 I copy all the files in /conf into my conf directory, /etc/hadoop, and then copy the minimal site configuration in= to . The advantage of using a directory that's not the /conf directory is = that it won't be overwritten the next time you untar a new build. Lastly, = I copy the minimal site configuration into the conf files. For the sake of= brevity, I won't include the properties in full xml format, but here are t= he ones I set: yarn-site.xml: yarn.nodemanager.aux-services =3D mapreduce.shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class =3D org.apache.hado= op.mapred.ShuffleHandler yarn.resourcemanager.scheduler.class =3D org.apache.hadoop.yarn.server.re= sourcemanager.scheduler.fair.FairScheduler mapred-site.xml: mapreduce.framework.name =3D yarn core-site.xml: fs.default.name =3D hdfs://localhost:9000 hdfs-site.xml: dfs.replication =3D 1 dfs.permissions =3D false Then, to format HDFS and start our cluster, we can simply do: ./hadoop-dev-setup.sh ./hadoop-dev.sh start To stop it: ./hadoop-dev.sh stop Once I have this set up, for quicker iteration, I have some scripts that bu= ild submodules (sometimes all of mapreduce, sometimes just the resourcemana= ger) and copy the updated jars into my setup. Regarding your last question, are you saying that you were able to load it = into Eclipse already, and want tips on the best way to browse within it? O= r that you're trying to get the source loaded into Eclipse? Hope that helps! Sandy On Thu, May 30, 2013 at 9:32 AM, John Lilley > wrote: Thanks for help me to build Hadoop! I'm through compile and install of mav= en plugins into Eclipse. I could use some pointers for next steps I want t= o take, which are: * Deploy the simplest "development only" cluster (single node?) and= learn how to debug within it. I read about the "local runner" configurati= on here (http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms), does t= hat still apply to MR2/YARN? It seems like an old page; perhaps there is a= newer FAQ? * Build and run the ApplicationMaster "shell" sample, and use that = as a starting point for a customer AM. I would much appreciate any advice = on getting the edit/build/debug cycle ironed out for an AM. * Setup Hadoop source for easier browsing and learning (Eclipse loa= d?). What is typically done to make for easy browsing of referenced classe= s/methods by name? Thanks John --_000_869970D71E26D7498BDAC4E1CA92226B658B75D3MBX021E3NJ2exch_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Sandy,<= /p>

Thanks for all of the tip= s, I will try this over the weekend.   Regarding the last questio= n, I am still trying to get the source loaded into Eclipse in a manner that facilitates easier browsing, symbol search, editing, etc.  Perha= ps I am just missing some obvious FAQ?  This is leading up to modifyin= g and debugging the “shell” ApplicationMaster sample.  Thi= s page:

ht= tp://stackoverflow.com/questions/11007423/developing-testing-and-debugging-= hadoop-map-reduce-jobs-with-eclipse

looks promising as a Hado= op-in-Eclipse strategy, but it is over a year old and I’m not sure if= it applies to Hadoop 2.0 and YARN.

John

 <= /p>

From: Sandy Ry= za [mailto:sandy.ryza@cloudera.com]
Sent: Friday, May 31, 2013 12:13 PM
To: user@hadoop.apache.org
Subject: Re: built hadoop! please help with next steps?

 

Hi John,

 

Here's how I deploy/debug Hadoop locally:=

To build and tar Hadoop:

 

  mvn clean package -Pdist -Dtar -DskipTests=3D= true

 

The tar will be located in the project directory und= er hadoop-dist/target/.  I untar it into my deploy directory.

 

I then copy these scripts into the same directory:

 

hadoop-dev-env.sh:

---

#!/bin/bash

export HADOOP_DEV_HOME=3D`pwd`

export HADOOP_MAPRED_HOME=3D${HADOOP_DEV_HOME}<= /o:p>

export HADOOP_COMMON_HOME=3D${HADOOP_DEV_HOME}<= /o:p>

export HADOOP_HDFS_HOME=3D${HADOOP_DEV_HOME}

export YARN_HOME=3D${HADOOP_DEV_HOME}

export HADOOP_CONF_DIR=3D${HADOOP_DEV_HOME}/etc/hado= op

 

hadoop-dev-setup.sh:

---

#!/bin/bash

source ./hadoop-dev-env.sh

bin/hadoop namenode -format

 

hadoop-dev.sh:

---

source hadoop-dev-env.sh

sbin/hadoop-daemon.sh $1 namenode

sbin/hadoop-daemon.sh $1 datanode

sbin/yarn-daemon.sh $1 resourcemanager

sbin/yarn-daemon.sh $1 nodemanager

sbin/mr-jobhistory-daemon.sh $1 historyserver

sbin/httpfs.sh $1

 

I copy all the files in <deploy directory>/con= f into my conf directory, <deploy directory>/etc/hadoop, and then cop= y the minimal site configuration into .  The advantage of using a dire= ctory that's not the /conf directory is that it won't be overwritten the next time you untar a new build.  Lastly, I copy t= he minimal site configuration into the conf files.  For the sake of br= evity, I won't include the properties in full xml format, but here are the = ones I set:

 

yarn-site.xml:

  yarn.nodemanager.aux-services =3D mapred= uce.shuffle

  yarn.nodemanager.aux-services.mapreduce.shuff= le.class =3D org.apache.hadoop.mapred.ShuffleHandler

  yarn.resourcemanager.scheduler.class =3D = ;org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler=

mapred-site.xml:

core-site.xml:

  fs.default= .name =3D hdfs://localhost:9000

hdfs-site.xml:

  dfs.replication =3D 1

  dfs.permissions =3D false

 

Then, to format HDFS and start our cluster, we can s= imply do:

./hadoop-dev-setup.sh

./hadoop-dev.sh start

To stop it:

./hadoop-dev.sh stop

 

Once I have this set up, for quicker iteration, I ha= ve some scripts that build submodules (sometimes all of mapreduce, sometime= s just the resourcemanager) and copy the updated jars into my setup.

 

Regarding your last question, are you saying that yo= u were able to load it into Eclipse already, and want tips on the best way = to browse within it?  Or that you're trying to get the source loaded i= nto Eclipse?

 

Hope that helps!

Sandy

On Thu, May 30, 2013 at 9:32 AM, John Lilley <john.lilley@redp= oint.net> wrote:

Thanks for help me to build Hadoop!  I’m through compil= e and install of maven plugins into Eclipse.  I could use some pointer= s for next steps I want to take, which are:

·         Deploy the simplest “development only” cluster (single n= ode?) and learn how to debug within it.  I read about the “local= runner” configuration here (http://wiki.apache.org/hado= op/HowToDebugMapReducePrograms), does that still apply to MR2/YARN?  It seems like an old page; perhap= s there is a newer FAQ?

·         Build and run the ApplicationMaster “shell” sample, and = use that as a starting point for a customer AM.  I would much apprecia= te any advice on getting the edit/build/debug cycle ironed out for an AM.

·         Setup Hadoop source for easier browsing and learning (Eclipse load?)= .  What is typically done to make for easy browsing of referenced clas= ses/methods by name?

 

Thanks

John

 

 

--_000_869970D71E26D7498BDAC4E1CA92226B658B75D3MBX021E3NJ2exch_--