Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:x-accept-language:
	mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding;
	b=b3LLWAC+FobyFMXUWt8w7xPZcgQaKnIbGl0142GvhqCDBIDGR8x6ddVSnJ3841JT
Message-ID: <467AC3D9.6000709@yahoo-inc.com>
Date: Thu, 21 Jun 2007 11:30:49 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla Thunderbird 1.0.6 (Windows/20050716)
MIME-Version: 1.0
To: hadoop-user@lucene.apache.org
Subject: Re: How to Start Hadoop Cluster from source code in Eclipse
References: <11229322.post@talk.nabble.com>
 <FEF96577B70B31488511FC73B2804144CA74E6@DEN-EXM-05.corp.ebay.com>
In-Reply-To: <FEF96577B70B31488511FC73B2804144CA74E6@DEN-EXM-05.corp.ebay.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I run entire one node cluster in eclipse by just executing main() (run 
or debug menus) for each component.
You need to configure eclipse correctly in order to do that. Can you 
compile the whole thing under eclipse?
NameNode example:
= Open NameNode.java in the editor.
= Run / Run
= New Java Application -> will create an entry under "Java Application" 
named NameNode
= Select NameNode, go to tab Arguments and enter the following arguments 
under "VM Arguments":
  -Dhadoop.log.dir=./logs  
  -Xmx500m
  -ea
    The first one is required, can point to your log directory, the 
other two are optional
= go to the "Classpath" tab, add "hadoop/build" path under "User entries" by
    Advanced / New Folder / select "hadoop/build"
That should be it, if the default classpath is configured correctly, and 
if I am not forgetting anything.
Let me know if that helped, I 'll send you screenshots of my 
configuration if not.

--Konstantin


Mahajan, Neeraj wrote:

>There are two sepearete issues you are asking here:
>1. How to modify/add to haddop code and execute the changed -
>Eclipse is just an IDE, it doesn't matter whether you use eclipse or
>some other editor.
>I have been using eclipse. What I do is modify the code using eclipse
>and then run "ant jar" in the root folder of hadoop (you could also
>configure this to work directly from eclipse). This would regenerate the
>jars and put them in build/ folder. Now you can either copy these jars
>into hadoop root folder (removing "dev" in their name) so that they
>replace the original jars or modify the scripts in bin/ to point to the
>newly generated jars.
>
>2. How to debug using a IDE -
>This page gives a high-level intro to debugging hadoop -
>http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms
>According to me, there are two ways you can debug hadoop programs: Run
>hadoop in local mode and debug in process in the IDE or run hadoop in
>distributed mode and remote debug using IDE.
>
>The first way is easy. In the bin/hadoop script at the end there is a
>exec command, instead of that put a echo command and run your program.
>You can see what the paramters the script passes while starting hadoop.
>Use these same parameters in the IDE and you can debug hadoop. Remember
>to make change to the conf files so that hadoop runs in local mode. To
>be more specific, you will have to set the program arguemnts, VM
>arguments and add an entry in the classpath pointing to the conf folder.
>
>The second method is compilcated. You will have to modify the scripts
>and put in some extra params like "-Xdebug
>-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=<port>" for the
>java command. Specify the <port> of you choice in it. On the server
>where you are running both the namenode/jobnode there will be a conflict
>as the same port would be specified. So you will have to do some
>intelligent scripting to take care of this. Once the java processes
>start you can attach eclipse debugger to that machine's <port> and set
>breakpoints. Till this part you can debug all the things before map
>reduce tasks. Mapp reduce tasks run in separate process, for debugging
>them you will have to figure out yourself.
>
>The best way is to debug using the first approach (as the above link
>says). I think by that approach you can fix any map-reduce related
>problems and for other purely distributed kind of problems you can
>follow the second approach.
>
>~ Neeraj
>
>-----Original Message-----
>From: KrzyCube [mailto:yuxh312@gmail.com] 
>Sent: Thursday, June 21, 2007 2:08 AM
>To: hadoop-user@lucene.apache.org
>Subject: How to Start Hadoop Cluster from source code in Eclipse
>
>
>Hi,all:
>
>I am using Eclipse to View Hadoop source code , and i want to trace to
>see how it works, I code a few code to call the FSClient  and when i
>call into the RPC  object, it can not to be deep more .
>
>So i just want to start cluster from source code , which i am holding
>them in Eclipse now. 
>I browse the start-*.sh , and find that it must start several threads ,
>such as namenode , datanode,secondnamenode. i just don't know how to
>figure out.
>
>or is there any way to attach my code to a running process , just as the
>gdb while we are debug c code 
>
>Does any body ever use Eclipse to debug these source code , please give
>some tip.
>
> 
>
>Thanks .
> 
>
>KrzyCube
>--
>View this message in context:
>http://www.nabble.com/How-to-Start-Hadoop-Cluster-from-source-code-in-Ec
>lipse-tf3957457.html#a11229322
>Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>  
>