incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Boyd <joseph.b...@cbsinteractive.com>
Subject Re: Debugging hama in eclipse
Date Wed, 22 Jun 2011 22:31:11 GMT
On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Instead removing 206 line, why don't we add dependencies as below?

I think removing line 206 is the right thing to do.  Hama doesn't
depend on commons-cli, hadoop-core does, and maven will properly pick
that up and include the jar if allowed.

The fact that Hama doesn't depend on the several items below was
correctly detected by maven, in the report description of HAMA-402.

You don't mention a particular problem with including the transitive
dependencies, but if your problem is that it includes too many items,
maven does have options to exclude a particular transitive dependency
on a case-by-case basis that might be more appropriate to use.

Including the dependencies below as you suggest puts Hama in the place
of maintaining a transitive dependency like commons-cli in lock-step
hadoop-core, where the actual dependency comes from.


This is all just my $.02 worth, I'm open to arguments in favor of
<excludeTransitive/>, and there is often more than one way to skin a
cat with respect to dependency management.


...joe



>    <dependency>
>      <groupId>org.mortbay.jetty</groupId>
>      <artifactId>jetty-annotations</artifactId>
>      <version>6.1H.22</version>
>    </dependency>
>    <dependency>
>      <groupId>org.slf4j</groupId>
>      <artifactId>slf4j-log4j12</artifactId>
>      <version>1.5.2</version>
>    </dependency>
>    <dependency>
>      <groupId>commons-cli</groupId>
>      <artifactId>commons-cli</artifactId>
>      <version>1.2</version>
>    </dependency>
>
> Thanks.
>
> On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> Oh... sorry, you're right.
>>
>> It looks like a bug. We have to fix this.
>>
>> [INFO] Total time: 4:36.098s
>> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>> [INFO] Final Memory: 20M/51M
>> [INFO] ------------------------------------------------------------------------
>> edward@test ~/workspace/hama-trunk $ ls lib/
>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>    zookeeper-3.3.1.jar
>> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>> -bash: bin/hama: Permission denied
>> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/commons/cli/ParseException
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.cli.ParseException
>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>        ... 3 more
>>
>>
>>
>> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
>>> That's so weird.
>>>
>>> It works well on my test cluster.
>>>
>>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>    zookeeper-3.3.1.jar
>>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>> Estimated value of PI is 3.1534
>>> Job Finished in 7.886 seconds
>>> edward@slave:~/workspace/hama-trunk$
>>>
>>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>> <joseph.boyd@cbsinteractive.com> wrote:
>>>> Replying to my own post ...
>>>>
>>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>> <joseph.boyd@cbsinteractive.com> wrote:
>>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <ashish.a77777@gmail.com>
wrote:
>>>>>>
>>>>>> When I try to run the hama daemons using the command -
>>>>>> bin/hama zookeeper
>>>>>>
>>>>>> I get this error -
>>>>
>>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>>                    <excludeTransitive>true</excludeTransitive>
>>>>
>>>> This prevents commons-cli (and other transitive dependencies) from
>>>> being copied to the lib dir like they should be.
>>>>
>>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>> JIRA for this after lunch, the command line is fairly broken with this
>>>> line in, especially after several dependencies were (correctly)
>>>> removed in HAMA-402.
>>>>
>>>>
>>>> ...joe
>>>>
>>>>
>>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>>> re-adding it fixes the NoClassDefFoundError.
>>>>>
>>>>> I believe commons-cli should have been pulled as part of the
>>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>>> pulled)
>>>>>
>>>>> While a quick work-around is to re-add the dependency here, the proper
>>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>>> depends on commons-cli.
>>>>>
>>>>>
>>>>> ...joe
>>>>>
>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/commons/cli/ParseException
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.commons.cli.ParseException
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>>> ... 3 more
>>>>>>
>>>>>>
>>>>>> I tried installing it a few times, but can't think of a solution
to the
>>>>>> problem. Can you please help ?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>> Sincerely,
>>>>>> Ashish
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I would suggest following steps to debug:
>>>>>>>
>>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>>>>
>>>>>>> % bin/hama zookeeper
>>>>>>> % bin/hama bspmaster
>>>>>>> % bin/hama groom
>>>>>>>
>>>>>>> 3. Use the logging API to log debug.
>>>>>>> For example, LOG.debug("like this");
>>>>>>>
>>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <ashish.a77777@gmail.com>
>>>>>>> wrote:
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I was wondering if anyone can give me instructions on how
to debug code
>>>>>>> in
>>>>>>> > distributed or pseudo-distributed mode ? It would really
help me.
>>>>>>> >
>>>>>>> > Thank you
>>>>>>> >
>>>>>>> > Sincerely,
>>>>>>> > Ashish
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>>>> >wrote:
>>>>>>> >
>>>>>>> >>
>>>>>>> >> Hi,
>>>>>>> >>
>>>>>>> >> I have written some code for my GSoC task (Hama-367)
and am now trying
>>>>>>> to
>>>>>>> >> debug.
>>>>>>> >>
>>>>>>> >> I am trying to set up the environment into pseudo-distributed
mode so I
>>>>>>> can
>>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml
has to
>>>>>>> be
>>>>>>> >> local as this means LocalBSPRunner (as localhost would
call 127.0.0.1
>>>>>>> >> server), but where do I specify that in the code in
eclipse ?
>>>>>>> >>
>>>>>>> >> My next question is - what is the difference between
MiniBSPCluster and
>>>>>>> >> LocalBSPRunner  and which file should I be using to
be able to step
>>>>>>> through
>>>>>>> >> the code ?
>>>>>>> >>
>>>>>>> >> These are the processes currently running on my machine
-
>>>>>>> >>
>>>>>>> >> 6112 NameNode
>>>>>>> >> 3319 ZooKeeperRunner
>>>>>>> >> 3018
>>>>>>> >> 6449 SecondaryNameNode
>>>>>>> >> 5709 HMaster
>>>>>>> >> 5643 HQuorumPeer
>>>>>>> >> 6684 TaskTracker
>>>>>>> >> 6526 JobTracker
>>>>>>> >> 8074 Jps
>>>>>>> >> 6277 DataNode
>>>>>>> >>
>>>>>>> >> Thank you,
>>>>>>> >> Ashish
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards, Edward J. Yoon
>>>>>>> @eddieyoon
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Mime
View raw message