incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Debugging hama in eclipse
Date Thu, 23 Jun 2011 08:12:32 GMT
You need to set the 'bsp.master.address' to localhost or your hostname.

conf/hama-site.xml:

  <property>
    <name>bsp.master.address</name>
    <value>localhost:40000</value>
  </property>

On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <ashish.a77777@gmail.com> wrote:
> Now, I am having a problem starting the bspmaster :(
> bin/hama bspmaster
>
>
> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
> 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting BSPMaster
> STARTUP_MSG:   host = ubuntu/127.0.1.1
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
> 13:22:54 PST 2010
> ************************************************************/
> 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
> 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
> java.net.UnknownHostException: Invalid hostname for server: local
>        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>
> 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
> ************************************************************/
> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>
>
> Thanks
> Ashish
>
> On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> If you want to execute a command in the background without watching
>> logs, you can do the following:
>>
>> % bin/hama zookeeper &
>>
>> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <ashish.a77777@gmail.com> wrote:
>>> How long does it take to execute the command
>>> bin/hama zookeeper
>>>
>>> I had to stop the execution as it stopped doing anything after some time.
>>>
>>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>> GMT
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:host.name=ubuntu.ubuntu-domain
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.version=1.6.0_24
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.vendor=Sun Microsystems Inc.
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.io.tmpdir=/tmp
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.compiler=<NA>
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:os.version=2.6.35-28-generic
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:user.name=hadoop
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:user.home=/home/hadoop
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>> 0.0.0.0/0.0.0.0:21810
>>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>
>>>
>>> Thanks
>>> Ashish
>>>
>>>
>>>
>>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>> <joseph.boyd@cbsinteractive.com> wrote:
>>>>
>>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <edwardyoon@apache.org>
wrote:
>>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>>
>>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>>> that up and include the jar if allowed.
>>>>
>>>> The fact that Hama doesn't depend on the several items below was
>>>> correctly detected by maven, in the report description of HAMA-402.
>>>>
>>>> You don't mention a particular problem with including the transitive
>>>> dependencies, but if your problem is that it includes too many items,
>>>> maven does have options to exclude a particular transitive dependency
>>>> on a case-by-case basis that might be more appropriate to use.
>>>>
>>>> Including the dependencies below as you suggest puts Hama in the place
>>>> of maintaining a transitive dependency like commons-cli in lock-step
>>>> hadoop-core, where the actual dependency comes from.
>>>>
>>>>
>>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>>> <excludeTransitive/>, and there is often more than one way to skin
a
>>>> cat with respect to dependency management.
>>>>
>>>>
>>>> ...joe
>>>>
>>>>
>>>>
>>>> >    <dependency>
>>>> >      <groupId>org.mortbay.jetty</groupId>
>>>> >      <artifactId>jetty-annotations</artifactId>
>>>> >      <version>6.1H.22</version>
>>>> >    </dependency>
>>>> >    <dependency>
>>>> >      <groupId>org.slf4j</groupId>
>>>> >      <artifactId>slf4j-log4j12</artifactId>
>>>> >      <version>1.5.2</version>
>>>> >    </dependency>
>>>> >    <dependency>
>>>> >      <groupId>commons-cli</groupId>
>>>> >      <artifactId>commons-cli</artifactId>
>>>> >      <version>1.2</version>
>>>> >    </dependency>
>>>> >
>>>> > Thanks.
>>>> >
>>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <edwardyoon@apache.org>
wrote:
>>>> >> Oh... sorry, you're right.
>>>> >>
>>>> >> It looks like a bug. We have to fix this.
>>>> >>
>>>> >> [INFO] Total time: 4:36.098s
>>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>>> >> [INFO] Final Memory: 20M/51M
>>>> >> [INFO] ------------------------------------------------------------------------
>>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>> >>    zookeeper-3.3.1.jar
>>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>>> >> -bash: bin/hama: Permission denied
>>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> >> org/apache/commons/cli/ParseException
>>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> >> Caused by: java.lang.ClassNotFoundException:
>>>> >> org.apache.commons.cli.ParseException
>>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>> >>        at java.security.AccessController.doPrivileged(Native
Method)
>>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>>> >>        ... 3 more
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <edwardyoon@apache.org>
wrote:
>>>> >>> That's so weird.
>>>> >>>
>>>> >>> It works well on my test cluster.
>>>> >>>
>>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>> >>>    zookeeper-3.3.1.jar
>>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps
number: 0
>>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps
number: 1
>>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of
supersteps: 1
>>>> >>> Estimated value of PI is 3.1534
>>>> >>> Job Finished in 7.886 seconds
>>>> >>> edward@slave:~/workspace/hama-trunk$
>>>> >>>
>>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>>> >>> <joseph.boyd@cbsinteractive.com> wrote:
>>>> >>>> Replying to my own post ...
>>>> >>>>
>>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>> >>>> <joseph.boyd@cbsinteractive.com> wrote:
>>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <ashish.a77777@gmail.com>
wrote:
>>>> >>>>>>
>>>> >>>>>> When I try to run the hama daemons using the command
-
>>>> >>>>>> bin/hama zookeeper
>>>> >>>>>>
>>>> >>>>>> I get this error -
>>>> >>>>
>>>> >>>> It looks like the problem is actually this line in pom.xml
(line 205)  :
>>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>>> >>>>
>>>> >>>> This prevents commons-cli (and other transitive dependencies)
from
>>>> >>>> being copied to the lib dir like they should be.
>>>> >>>>
>>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.
 I can open a
>>>> >>>> JIRA for this after lunch, the command line is fairly broken
with this
>>>> >>>> line in, especially after several dependencies were (correctly)
>>>> >>>> removed in HAMA-402.
>>>> >>>>
>>>> >>>>
>>>> >>>> ...joe
>>>> >>>>
>>>> >>>>
>>>> >>>>> I'm getting the same NoClassDefFoundError using the
svn code.  The
>>>> >>>>> commons-cli dependency was removed recently as part
of HAMA-402, and
>>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>>> >>>>>
>>>> >>>>> I believe commons-cli should have been pulled as part
of the
>>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven
skills are
>>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies
are
>>>> >>>>> pulled)
>>>> >>>>>
>>>> >>>>> While a quick work-around is to re-add the dependency
here, the proper
>>>> >>>>> fix is probably to fix the hadoop-core pom, and properly
mark that it
>>>> >>>>> depends on commons-cli.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> ...joe
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> >>>>>> org/apache/commons/cli/ParseException
>>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>>> >>>>>> org.apache.commons.cli.ParseException
>>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> >>>>>> at java.security.AccessController.doPrivileged(Native
Method)
>>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> >>>>>> ... 3 more
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I tried installing it a few times, but can't think
of a solution to the
>>>> >>>>>> problem. Can you please help ?
>>>> >>>>>>
>>>> >>>>>> Thank you
>>>> >>>>>>
>>>> >>>>>> Sincerely,
>>>> >>>>>> Ashish
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon
<edwardyoon@apache.org>wrote:
>>>> >>>>>>
>>>> >>>>>>> Hi,
>>>> >>>>>>>
>>>> >>>>>>> I would suggest following steps to debug:
>>>> >>>>>>>
>>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>> >>>>>>>
>>>> >>>>>>> % bin/hama zookeeper
>>>> >>>>>>> % bin/hama bspmaster
>>>> >>>>>>> % bin/hama groom
>>>> >>>>>>>
>>>> >>>>>>> 3. Use the logging API to log debug.
>>>> >>>>>>> For example, LOG.debug("like this");
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal
<ashish.a77777@gmail.com>
>>>> >>>>>>> wrote:
>>>> >>>>>>> > Hi,
>>>> >>>>>>> >
>>>> >>>>>>> > I was wondering if anyone can give me instructions
on how to debug code
>>>> >>>>>>> in
>>>> >>>>>>> > distributed or pseudo-distributed mode
? It would really help me.
>>>> >>>>>>> >
>>>> >>>>>>> > Thank you
>>>> >>>>>>> >
>>>> >>>>>>> > Sincerely,
>>>> >>>>>>> > Ashish
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish
Agarwal <ashish.a77777@gmail.com
>>>> >>>>>>> >wrote:
>>>> >>>>>>> >
>>>> >>>>>>> >>
>>>> >>>>>>> >> Hi,
>>>> >>>>>>> >>
>>>> >>>>>>> >> I have written some code for my GSoC
task (Hama-367) and am now trying
>>>> >>>>>>> to
>>>> >>>>>>> >> debug.
>>>> >>>>>>> >>
>>>> >>>>>>> >> I am trying to set up the environment
into pseudo-distributed mode so I
>>>> >>>>>>> can
>>>> >>>>>>> >> debug. I think the value of bsp.master.address
in hama-site.xml has to
>>>> >>>>>>> be
>>>> >>>>>>> >> local as this means LocalBSPRunner
(as localhost would call 127.0.0.1
>>>> >>>>>>> >> server), but where do I specify that
in the code in eclipse ?
>>>> >>>>>>> >>
>>>> >>>>>>> >> My next question is - what is the difference
between MiniBSPCluster and
>>>> >>>>>>> >> LocalBSPRunner  and which file should
I be using to be able to step
>>>> >>>>>>> through
>>>> >>>>>>> >> the code ?
>>>> >>>>>>> >>
>>>> >>>>>>> >> These are the processes currently running
on my machine -
>>>> >>>>>>> >>
>>>> >>>>>>> >> 6112 NameNode
>>>> >>>>>>> >> 3319 ZooKeeperRunner
>>>> >>>>>>> >> 3018
>>>> >>>>>>> >> 6449 SecondaryNameNode
>>>> >>>>>>> >> 5709 HMaster
>>>> >>>>>>> >> 5643 HQuorumPeer
>>>> >>>>>>> >> 6684 TaskTracker
>>>> >>>>>>> >> 6526 JobTracker
>>>> >>>>>>> >> 8074 Jps
>>>> >>>>>>> >> 6277 DataNode
>>>> >>>>>>> >>
>>>> >>>>>>> >> Thank you,
>>>> >>>>>>> >> Ashish
>>>> >>>>>>> >>
>>>> >>>>>>> >
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> --
>>>> >>>>>>> Best Regards, Edward J. Yoon
>>>> >>>>>>> @eddieyoon
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Best Regards, Edward J. Yoon
>>>> >>> @eddieyoon
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards, Edward J. Yoon
>>>> >> @eddieyoon
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Best Regards, Edward J. Yoon
>>>> > @eddieyoon
>>>> >
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message