incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Agarwal <ashish.a77...@gmail.com>
Subject Re: Debugging hama in eclipse
Date Thu, 23 Jun 2011 07:43:11 GMT
How long does it take to execute the command
bin/hama zookeeper

I had to stop the execution as it stopped doing anything after some time.

hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
GMT
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:host.name=ubuntu.ubuntu-domain
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.version=1.6.0_24
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.vendor=Sun Microsystems Inc.
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.io.tmpdir=/tmp
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.compiler=<NA>
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:os.version=2.6.35-28-generic
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:user.name=hadoop
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:user.home=/home/hadoop
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
0.0.0.0/0.0.0.0:21810
11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
/tmp/hama-hadoop/zookeeper/version-2/snapshot.0
11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
/tmp/hama-hadoop/zookeeper/version-2/snapshot.0
11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$


Thanks
Ashish



On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
<joseph.boyd@cbsinteractive.com> wrote:
>
> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> > Instead removing 206 line, why don't we add dependencies as below?
>
> I think removing line 206 is the right thing to do.  Hama doesn't
> depend on commons-cli, hadoop-core does, and maven will properly pick
> that up and include the jar if allowed.
>
> The fact that Hama doesn't depend on the several items below was
> correctly detected by maven, in the report description of HAMA-402.
>
> You don't mention a particular problem with including the transitive
> dependencies, but if your problem is that it includes too many items,
> maven does have options to exclude a particular transitive dependency
> on a case-by-case basis that might be more appropriate to use.
>
> Including the dependencies below as you suggest puts Hama in the place
> of maintaining a transitive dependency like commons-cli in lock-step
> hadoop-core, where the actual dependency comes from.
>
>
> This is all just my $.02 worth, I'm open to arguments in favor of
> <excludeTransitive/>, and there is often more than one way to skin a
> cat with respect to dependency management.
>
>
> ...joe
>
>
>
> >    <dependency>
> >      <groupId>org.mortbay.jetty</groupId>
> >      <artifactId>jetty-annotations</artifactId>
> >      <version>6.1H.22</version>
> >    </dependency>
> >    <dependency>
> >      <groupId>org.slf4j</groupId>
> >      <artifactId>slf4j-log4j12</artifactId>
> >      <version>1.5.2</version>
> >    </dependency>
> >    <dependency>
> >      <groupId>commons-cli</groupId>
> >      <artifactId>commons-cli</artifactId>
> >      <version>1.2</version>
> >    </dependency>
> >
> > Thanks.
> >
> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> >> Oh... sorry, you're right.
> >>
> >> It looks like a bug. We have to fix this.
> >>
> >> [INFO] Total time: 4:36.098s
> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
> >> [INFO] Final Memory: 20M/51M
> >> [INFO] ------------------------------------------------------------------------
> >> edward@test ~/workspace/hama-trunk $ ls lib/
> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
> >>    zookeeper-3.3.1.jar
> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
> >> -bash: bin/hama: Permission denied
> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> org/apache/commons/cli/ParseException
> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.commons.cli.ParseException
> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >>        at java.security.AccessController.doPrivileged(Native Method)
> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> >>        ... 3 more
> >>
> >>
> >>
> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <edwardyoon@apache.org>
wrote:
> >>> That's so weird.
> >>>
> >>> It works well on my test cluster.
> >>>
> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
> >>>    zookeeper-3.3.1.jar
> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps:
1
> >>> Estimated value of PI is 3.1534
> >>> Job Finished in 7.886 seconds
> >>> edward@slave:~/workspace/hama-trunk$
> >>>
> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
> >>> <joseph.boyd@cbsinteractive.com> wrote:
> >>>> Replying to my own post ...
> >>>>
> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
> >>>> <joseph.boyd@cbsinteractive.com> wrote:
> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <ashish.a77777@gmail.com>
wrote:
> >>>>>>
> >>>>>> When I try to run the hama daemons using the command -
> >>>>>> bin/hama zookeeper
> >>>>>>
> >>>>>> I get this error -
> >>>>
> >>>> It looks like the problem is actually this line in pom.xml (line 205)
 :
> >>>>                    <excludeTransitive>true</excludeTransitive>
> >>>>
> >>>> This prevents commons-cli (and other transitive dependencies) from
> >>>> being copied to the lib dir like they should be.
> >>>>
> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open
a
> >>>> JIRA for this after lunch, the command line is fairly broken with this
> >>>> line in, especially after several dependencies were (correctly)
> >>>> removed in HAMA-402.
> >>>>
> >>>>
> >>>> ...joe
> >>>>
> >>>>
> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
> >>>>> commons-cli dependency was removed recently as part of HAMA-402,
and
> >>>>> re-adding it fixes the NoClassDefFoundError.
> >>>>>
> >>>>> I believe commons-cli should have been pulled as part of the
> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
> >>>>> pulled)
> >>>>>
> >>>>> While a quick work-around is to re-add the dependency here, the
proper
> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that
it
> >>>>> depends on commons-cli.
> >>>>>
> >>>>>
> >>>>> ...joe
> >>>>>
> >>>>>
> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>>>> org/apache/commons/cli/ParseException
> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> >>>>>> Caused by: java.lang.ClassNotFoundException:
> >>>>>> org.apache.commons.cli.ParseException
> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >>>>>> ... 3 more
> >>>>>>
> >>>>>>
> >>>>>> I tried installing it a few times, but can't think of a solution
to the
> >>>>>> problem. Can you please help ?
> >>>>>>
> >>>>>> Thank you
> >>>>>>
> >>>>>> Sincerely,
> >>>>>> Ashish
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I would suggest following steps to debug:
> >>>>>>>
> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
> >>>>>>> 2. Run the Hama deamons with 3 terminals.
> >>>>>>>
> >>>>>>> % bin/hama zookeeper
> >>>>>>> % bin/hama bspmaster
> >>>>>>> % bin/hama groom
> >>>>>>>
> >>>>>>> 3. Use the logging API to log debug.
> >>>>>>> For example, LOG.debug("like this");
> >>>>>>>
> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <ashish.a77777@gmail.com>
> >>>>>>> wrote:
> >>>>>>> > Hi,
> >>>>>>> >
> >>>>>>> > I was wondering if anyone can give me instructions
on how to debug code
> >>>>>>> in
> >>>>>>> > distributed or pseudo-distributed mode ? It would really
help me.
> >>>>>>> >
> >>>>>>> > Thank you
> >>>>>>> >
> >>>>>>> > Sincerely,
> >>>>>>> > Ashish
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
> >>>>>>> >wrote:
> >>>>>>> >
> >>>>>>> >>
> >>>>>>> >> Hi,
> >>>>>>> >>
> >>>>>>> >> I have written some code for my GSoC task (Hama-367)
and am now trying
> >>>>>>> to
> >>>>>>> >> debug.
> >>>>>>> >>
> >>>>>>> >> I am trying to set up the environment into pseudo-distributed
mode so I
> >>>>>>> can
> >>>>>>> >> debug. I think the value of bsp.master.address
in hama-site.xml has to
> >>>>>>> be
> >>>>>>> >> local as this means LocalBSPRunner (as localhost
would call 127.0.0.1
> >>>>>>> >> server), but where do I specify that in the code
in eclipse ?
> >>>>>>> >>
> >>>>>>> >> My next question is - what is the difference between
MiniBSPCluster and
> >>>>>>> >> LocalBSPRunner  and which file should I be using
to be able to step
> >>>>>>> through
> >>>>>>> >> the code ?
> >>>>>>> >>
> >>>>>>> >> These are the processes currently running on my
machine -
> >>>>>>> >>
> >>>>>>> >> 6112 NameNode
> >>>>>>> >> 3319 ZooKeeperRunner
> >>>>>>> >> 3018
> >>>>>>> >> 6449 SecondaryNameNode
> >>>>>>> >> 5709 HMaster
> >>>>>>> >> 5643 HQuorumPeer
> >>>>>>> >> 6684 TaskTracker
> >>>>>>> >> 6526 JobTracker
> >>>>>>> >> 8074 Jps
> >>>>>>> >> 6277 DataNode
> >>>>>>> >>
> >>>>>>> >> Thank you,
> >>>>>>> >> Ashish
> >>>>>>> >>
> >>>>>>> >
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Best Regards, Edward J. Yoon
> >>>>>>> @eddieyoon
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon
> >>> @eddieyoon
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >

Mime
View raw message