hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@googlemail.com>
Subject Re: Eclipse test failure for HAMA-376
Date Fri, 15 Apr 2011 07:50:48 GMT
I would like to have these configurations a tiny bit like Hadoop. Cause it's
proven and easy.
We should distinguish between a person how runs eclipse and want to develop
a BSP, real cluster with n-slaves and a (pseudo-distributed) one machine
cluster.

Most likely when a user downloads a hama release he will setup a
pseudo-distributed environment on his linux machine.
This is just a shortcut version of a cluster / distributed version. Let him
submit some examples and look whether he needs it in his production
environment.
If he decided to do so, he maybe wants to setup a distributed environment.
BUT he really needs to test his own BSP and develop it. It is not easy if
you are not familiar with the framework.
The LocalBSPJobRunner is just the same as a LocalJobRunner in Hadoop (this
was added for testing mapreduce stuff on your local machine AND especially
on windows, without submitting anything to a production environment or
running deamons). Just for stepping through the code and getting familiar
with the framework.

Nobody will have a conf directory in his own trunk when he develops for a
company with hama.
Most of them will setup a management utitily where configurations are stored
and jobs are submitted, from time to time with a scheduler.

Short summary of my opinion

   1. User running on linux with hama deamons should receive the full
   cluster experience (just the way it is)
   2. User running on eclipse/other IDE should receive a full debug version
   that emulates a cluster through multithreading (localjobrunner)

Therefore we should not add anything strange to bsp.master.address. Just let
this be localhost.

greetings from germany

2011/4/15 Edward J. Yoon <edwardyoon@apache.org>

> I think, many things should be fixed.
>
> 1. Basically the value of "bsp.master.address" is setted to
> "localhost" In hama-default.xml:
>
>    <name>bsp.master.address</name>
>    <value>localhost</value>
>    <description>The address of the bsp master server. Either the
>    literal string "local" or a host[:port] (where host is a name or
>
> If we add a LocalBSPJobRunner to Hama 0.3, I think, we should use
> "local" as a default value. ("local" ≠ "localhost"). Description
> strings should be fixed, too.
>
> 2. In eclipse, conf.get("bsp.master.address") returns null.
>
>    String mode = conf.get("bsp.master.address");
>    LOG.info(">>>>>> " + mode);
>
> It means that the config files are not loaded. If we run with advanced
> configurations in eclipse, it'll be work but, it's really annoying.
>
> I think, we can set the basic configurations for JUnit tests by
> hard-coding in MiniClusterTest constructor.
>
> On Thu, Apr 14, 2011 at 8:02 PM, chl501 <chl501@nuk.edu.tw> wrote:
> > My question should be `Can't we simply use the default value, which sets
> bsp.master.address to localhost, in hama-default.xml'?
> >
> > Currently when using the default value (localhost), the bspMasterAddr
> will has the value referenced to master address/port.
> >
> > Resetting bsp.master.address to local will cause bspMasterAddr to null.
> >
> > Why do we need to reset bsp.master.address to local?
> >
> > Sorry for my poor English.
> >
> > -----Original message-----
> > From:chl501 <chl501@nuk.edu.tw>
> > To:hama-dev <hama-dev@incubator.apache.org>
> > Date:Thu, 14 Apr 2011 17:45:38 +0800 (CST)
> > Subject:Re: Eclipse test failure for HAMA-376
> >
> > Looks like the problem is because BSPMaster.getAddress(conf) in
> GroomServer is trying to create socket again after BSPMaster successfully
> occupies the ip/port (GroomServers need to wait until BSPMaster is up).
> >
> > in the end of NetUtil.createSocketAddr() it shows
> >
> > public static InetSocketAddress createSocketAddr(String target,
> >                                                   int defaultPort) {
> >    ...
> >    return new InetSocketAddress(hostname, port);
> > }
> >
> > Should we change not to use bspMasterAddr because GroomServer can
> directly obtain bsp.master.address (ip/port) from conf/hama-site.xml?
> >
> > -----Original message-----
> > From:Edward J. Yoon <edwardyoon@apache.org>
> > To:hama-dev@incubator.apache.org,chl501@nuk.edu.tw
> > Date:Thu, 14 Apr 2011 17:32:01 +0900
> > Subject:Re: Eclipse test failure for HAMA-376
> >
> > Nope, "Class not found error" and "setJar()" problems are unconnected
> > things from my report.
> >
> > I think, the point of miniBSPCluster problem with eclipse and maven
> > build, is that the value of "bsp.master.address" is always null:
> >
> >>11/04/12 20:49:50 INFO bsp.GroomServer: groom start
> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >
> > Try to set the value of "mode" to "local" as below:
> >
> >  public GroomServer(Configuration conf) throws IOException {
> >    ....
> >    mode = "local";
> >    ....
> >
> > 2011/4/14 chl501 <chl501@nuk.edu.tw>:
> >> It is the same issue that the jar file is missing. If it is required to
> ask use explicitly setJar, is there any place that we can reference to the
> build version? For instance, the BSPJob.setJar() requires to set jar path
> e.g. /path/to/eclipse/hama-1090935/build/hama-0.3.0-test.jar where build
> version 0.3.0 is changed according to ant's property ${version}. Otherwise,
> we may need to parse ant file for explicitly adding this info.
> >>
> >> 11/04/14 14:50:13 WARN bsp.BSPJobClient: No job jar file set.  User
> classes may not be found. See BSPJob#setJar(String) or check Your jar file.
> >> 11/04/14 14:50:13 INFO bsp.GroomServer: jobConf.getJar():null
> >> 11/04/14 14:50:13 WARN bsp.GroomServer: Error initializing
> attempt_201104141450_0001_000001_0:
> >> java.lang.IllegalArgumentException: Can not create a Path from a null
> string
> >>        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> >>        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
> >>        at
> org.apache.hama.bsp.GroomServer.localizeJob(GroomServer.java:519)
> >>        at
> org.apache.hama.bsp.GroomServer.startNewTask(GroomServer.java:451)
> >>        at
> org.apache.hama.bsp.GroomServer.access$100(GroomServer.java:70)
> >>        at
> org.apache.hama.bsp.GroomServer$DispatchTasksHandler.handle(GroomServer.java:144)
> >>        at
> org.apache.hama.bsp.GroomServer$Instructor.run(GroomServer.java:192)
> >>
> >>
> >>>Basically, your miniBSPCluster doesn't work with eclipse JUnit test and
> maven build as below:
> >>
> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Cleaning up the system directory
> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: file:/tmp/hadoop/bsp/system
> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: java.lang.NullPointerException
> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server Responder: starting
> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server listener on 40000:
> starting
> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server handler 0 on 40000:
> starting
> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Starting RUNNING
> >>>11/04/12 20:49:50 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:50 INFO bsp.GroomServer: groom start
> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>11/04/12 20:49:56 INFO hama.MiniBSPCluster: Waitin for GroomServer up.
> >>>^CDestroying 1 processes
> >>>Destroying process..
> >>>Destroyed 1 processes
> >>>Can you find out the reason why?
> >>
> >>
> >>
> >>
> >>
> >> ChiaHung Lin
> >> Department of Information Management
> >> National University of Kaohsiung
> >> Taiwan
> >>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > http://blog.udanax.org
> > http://twitter.com/eddieyoon
> >
> >
> > ChiaHung Lin
> > Department of Information Management
> > National University of Kaohsiung
> > Taiwan
> >
> >
> > ChiaHung Lin
> > Department of Information Management
> > National University of Kaohsiung
> > Taiwan
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> http://blog.udanax.org
> http://twitter.com/eddieyoon
>



-- 
Thomas Jungblut
Berlin

mobile: 0170-3081070

business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message