incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Eclipse test failure for HAMA-376
Date Fri, 15 Apr 2011 08:38:06 GMT
Again, ..

1. JUnit tests should be run with maven build and eclipse. * eclipse
is not a point. Please see my comment again:

https://issues.apache.org/jira/browse/HAMA-376?focusedCommentId=13018799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13018799

2. By default ("local"), user should able to run the BSP job without
starting any daemons. It's not for unit tests.

 - "local" uses LocalBSPJobRunner.
 - "localhost" means that the '127.0.0.1' server.

In hadoop case, by default, you can run the examples using
LocalJobRunner without starting hadoop-deamons.

...
edward@edward-VirtualBox:~/workspace/hadoop-0.18$ ps -ef | grep java
edward    3137  1881  0 17:23 pts/0    00:00:00 grep --color=auto java

edward@edward-VirtualBox:~/workspace/hadoop-0.18$ bin/hadoop jar
build/hadoop-0.18.4-dev-examples.jar pi 1 1
Number of Maps = 1 Samples per Map = 1
Wrote input for Map #0
Starting Job
11/04/15 17:23:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
11/04/15 17:23:28 INFO mapred.JobClient: Running job: job_local_0001
11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1



On Fri, Apr 15, 2011 at 4:50 PM, Thomas Jungblut
<thomas.jungblut@googlemail.com> wrote:
> I would like to have these configurations a tiny bit like Hadoop. Cause it's
> proven and easy.
> We should distinguish between a person how runs eclipse and want to develop
> a BSP, real cluster with n-slaves and a (pseudo-distributed) one machine
> cluster.
>
> Most likely when a user downloads a hama release he will setup a
> pseudo-distributed environment on his linux machine.
> This is just a shortcut version of a cluster / distributed version. Let him
> submit some examples and look whether he needs it in his production
> environment.
> If he decided to do so, he maybe wants to setup a distributed environment.
> BUT he really needs to test his own BSP and develop it. It is not easy if
> you are not familiar with the framework.
> The LocalBSPJobRunner is just the same as a LocalJobRunner in Hadoop (this
> was added for testing mapreduce stuff on your local machine AND especially
> on windows, without submitting anything to a production environment or
> running deamons). Just for stepping through the code and getting familiar
> with the framework.
>
> Nobody will have a conf directory in his own trunk when he develops for a
> company with hama.
> Most of them will setup a management utitily where configurations are stored
> and jobs are submitted, from time to time with a scheduler.
>
> Short summary of my opinion
>
>   1. User running on linux with hama deamons should receive the full
>   cluster experience (just the way it is)
>   2. User running on eclipse/other IDE should receive a full debug version
>   that emulates a cluster through multithreading (localjobrunner)
>
> Therefore we should not add anything strange to bsp.master.address. Just let
> this be localhost.
>
> greetings from germany
>
> 2011/4/15 Edward J. Yoon <edwardyoon@apache.org>
>
>> I think, many things should be fixed.
>>
>> 1. Basically the value of "bsp.master.address" is setted to
>> "localhost" In hama-default.xml:
>>
>>    <name>bsp.master.address</name>
>>    <value>localhost</value>
>>    <description>The address of the bsp master server. Either the
>>    literal string "local" or a host[:port] (where host is a name or
>>
>> If we add a LocalBSPJobRunner to Hama 0.3, I think, we should use
>> "local" as a default value. ("local" ≠ "localhost"). Description
>> strings should be fixed, too.
>>
>> 2. In eclipse, conf.get("bsp.master.address") returns null.
>>
>>    String mode = conf.get("bsp.master.address");
>>    LOG.info(">>>>>> " + mode);
>>
>> It means that the config files are not loaded. If we run with advanced
>> configurations in eclipse, it'll be work but, it's really annoying.
>>
>> I think, we can set the basic configurations for JUnit tests by
>> hard-coding in MiniClusterTest constructor.
>>
>> On Thu, Apr 14, 2011 at 8:02 PM, chl501 <chl501@nuk.edu.tw> wrote:
>> > My question should be `Can't we simply use the default value, which sets
>> bsp.master.address to localhost, in hama-default.xml'?
>> >
>> > Currently when using the default value (localhost), the bspMasterAddr
>> will has the value referenced to master address/port.
>> >
>> > Resetting bsp.master.address to local will cause bspMasterAddr to null.
>> >
>> > Why do we need to reset bsp.master.address to local?
>> >
>> > Sorry for my poor English.
>> >
>> > -----Original message-----
>> > From:chl501 <chl501@nuk.edu.tw>
>> > To:hama-dev <hama-dev@incubator.apache.org>
>> > Date:Thu, 14 Apr 2011 17:45:38 +0800 (CST)
>> > Subject:Re: Eclipse test failure for HAMA-376
>> >
>> > Looks like the problem is because BSPMaster.getAddress(conf) in
>> GroomServer is trying to create socket again after BSPMaster successfully
>> occupies the ip/port (GroomServers need to wait until BSPMaster is up).
>> >
>> > in the end of NetUtil.createSocketAddr() it shows
>> >
>> > public static InetSocketAddress createSocketAddr(String target,
>> >                                                   int
defaultPort) {
>> >    ...
>> >    return new InetSocketAddress(hostname, port);
>> > }
>> >
>> > Should we change not to use bspMasterAddr because GroomServer can
>> directly obtain bsp.master.address (ip/port) from conf/hama-site.xml?
>> >
>> > -----Original message-----
>> > From:Edward J. Yoon <edwardyoon@apache.org>
>> > To:hama-dev@incubator.apache.org,chl501@nuk.edu.tw
>> > Date:Thu, 14 Apr 2011 17:32:01 +0900
>> > Subject:Re: Eclipse test failure for HAMA-376
>> >
>> > Nope, "Class not found error" and "setJar()" problems are unconnected
>> > things from my report.
>> >
>> > I think, the point of miniBSPCluster problem with eclipse and maven
>> > build, is that the value of "bsp.master.address" is always null:
>> >
>> >>11/04/12 20:49:50 INFO bsp.GroomServer: groom start
>> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >
>> > Try to set the value of "mode" to "local" as below:
>> >
>> >  public GroomServer(Configuration conf) throws IOException {
>> >    ....
>> >    mode = "local";
>> >    ....
>> >
>> > 2011/4/14 chl501 <chl501@nuk.edu.tw>:
>> >> It is the same issue that the jar file is missing. If it is required to
>> ask use explicitly setJar, is there any place that we can reference to the
>> build version? For instance, the BSPJob.setJar() requires to set jar path
>> e.g. /path/to/eclipse/hama-1090935/build/hama-0.3.0-test.jar where build
>> version 0.3.0 is changed according to ant's property ${version}. Otherwise,
>> we may need to parse ant file for explicitly adding this info.
>> >>
>> >> 11/04/14 14:50:13 WARN bsp.BSPJobClient: No job jar file set.  User
>> classes may not be found. See BSPJob#setJar(String) or check Your jar file.
>> >> 11/04/14 14:50:13 INFO bsp.GroomServer: jobConf.getJar():null
>> >> 11/04/14 14:50:13 WARN bsp.GroomServer: Error initializing
>> attempt_201104141450_0001_000001_0:
>> >> java.lang.IllegalArgumentException: Can not create a Path from a null
>> string
>> >>        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
>> >>        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>> >>        at
>> org.apache.hama.bsp.GroomServer.localizeJob(GroomServer.java:519)
>> >>        at
>> org.apache.hama.bsp.GroomServer.startNewTask(GroomServer.java:451)
>> >>        at
>> org.apache.hama.bsp.GroomServer.access$100(GroomServer.java:70)
>> >>        at
>> org.apache.hama.bsp.GroomServer$DispatchTasksHandler.handle(GroomServer.java:144)
>> >>        at
>> org.apache.hama.bsp.GroomServer$Instructor.run(GroomServer.java:192)
>> >>
>> >>
>> >>>Basically, your miniBSPCluster doesn't work with eclipse JUnit test and
>> maven build as below:
>> >>
>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Cleaning up the system directory
>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: file:/tmp/hadoop/bsp/system
>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: java.lang.NullPointerException
>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server Responder: starting
>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server listener on 40000:
>> starting
>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server handler 0 on 40000:
>> starting
>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Starting RUNNING
>> >>>11/04/12 20:49:50 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:50 INFO bsp.GroomServer: groom start
>> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>11/04/12 20:49:56 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
>> >>>^CDestroying 1 processes
>> >>>Destroying process..
>> >>>Destroyed 1 processes
>> >>>Can you find out the reason why?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ChiaHung Lin
>> >> Department of Information Management
>> >> National University of Kaohsiung
>> >> Taiwan
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > http://blog.udanax.org
>> > http://twitter.com/eddieyoon
>> >
>> >
>> > ChiaHung Lin
>> > Department of Information Management
>> > National University of Kaohsiung
>> > Taiwan
>> >
>> >
>> > ChiaHung Lin
>> > Department of Information Management
>> > National University of Kaohsiung
>> > Taiwan
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> http://blog.udanax.org
>> http://twitter.com/eddieyoon
>>
>
>
>
> --
> Thomas Jungblut
> Berlin
>
> mobile: 0170-3081070
>
> business: thomas.jungblut@testberichte.de
> private: thomas.jungblut@gmail.com
>



-- 
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon

Mime
View raw message