incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chl501" <chl...@nuk.edu.tw>
Subject Re: Eclipse test failure for HAMA-376
Date Mon, 18 Apr 2011 06:17:12 GMT

Thanks for the explain. 

Also, a separate ticket is created for setJar issue. 

-----Original message-----
From:Edward J. Yoon <edwardyoon@apache.org>
To:hama-dev@incubator.apache.org
Date:Fri, 15 Apr 2011 17:56:47 +0900
Subject:Re: Eclipse test failure for HAMA-376

P.S., It would be nice if we can separate setJar() problem into another ticket!

On Fri, Apr 15, 2011 at 5:38 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Again, ..
>
> 1. JUnit tests should be run with maven build and eclipse. * eclipse
> is not a point. Please see my comment again:
>
> https://issues.apache.org/jira/browse/HAMA-376?focusedCommentId=13018799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13018799
>
> 2. By default ("local"), user should able to run the BSP job without
> starting any daemons. It's not for unit tests.
>
>  - "local" uses LocalBSPJobRunner.
>  - "localhost" means that the '127.0.0.1' server.
>
> In hadoop case, by default, you can run the examples using
> LocalJobRunner without starting hadoop-deamons.
>
> ...
> edward@edward-VirtualBox:~/workspace/hadoop-0.18$ ps -ef | grep java
> edward    3137  1881  0 17:23 pts/0    00:00:00 grep --color=auto java
>
> edward@edward-VirtualBox:~/workspace/hadoop-0.18$ bin/hadoop jar
> build/hadoop-0.18.4-dev-examples.jar pi 1 1
> Number of Maps = 1 Samples per Map = 1
> Wrote input for Map #0
> Starting Job
> 11/04/15 17:23:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
> 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
> 11/04/15 17:23:28 INFO mapred.JobClient: Running job: job_local_0001
> 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
> 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1
>
>
>
> On Fri, Apr 15, 2011 at 4:50 PM, Thomas Jungblut
> <thomas.jungblut@googlemail.com> wrote:
>> I would like to have these configurations a tiny bit like Hadoop. Cause it's
>> proven and easy.
>> We should distinguish between a person how runs eclipse and want to develop
>> a BSP, real cluster with n-slaves and a (pseudo-distributed) one machine
>> cluster.
>>
>> Most likely when a user downloads a hama release he will setup a
>> pseudo-distributed environment on his linux machine.
>> This is just a shortcut version of a cluster / distributed version. Let him
>> submit some examples and look whether he needs it in his production
>> environment.
>> If he decided to do so, he maybe wants to setup a distributed environment.
>> BUT he really needs to test his own BSP and develop it. It is not easy if
>> you are not familiar with the framework.
>> The LocalBSPJobRunner is just the same as a LocalJobRunner in Hadoop (this
>> was added for testing mapreduce stuff on your local machine AND especially
>> on windows, without submitting anything to a production environment or
>> running deamons). Just for stepping through the code and getting familiar
>> with the framework.
>>
>> Nobody will have a conf directory in his own trunk when he develops for a
>> company with hama.
>> Most of them will setup a management utitily where configurations are stored
>> and jobs are submitted, from time to time with a scheduler.
>>
>> Short summary of my opinion
>>
>>   1. User running on linux with hama deamons should receive the full
>>   cluster experience (just the way it is)
>>   2. User running on eclipse/other IDE should receive a full debug version
>>   that emulates a cluster through multithreading (localjobrunner)
>>
>> Therefore we should not add anything strange to bsp.master.address. Just let
>> this be localhost.
>>
>> greetings from germany
>>
>> 2011/4/15 Edward J. Yoon <edwardyoon@apache.org>
>>
>>> I think, many things should be fixed.
>>>
>>> 1. Basically the value of "bsp.master.address" is setted to
>>> "localhost" In hama-default.xml:
>>>
>>>    <name>bsp.master.address</name>
>>>    <value>localhost</value>
>>>    <description>The address of the bsp master server. Either the
>>>    literal string "local" or a host[:port] (where host is a name or
>>>
>>> If we add a LocalBSPJobRunner to Hama 0.3, I think, we should use
>>> "local" as a default value. ("local" ≠ "localhost"). Description
>>> strings should be fixed, too.
>>>
>>> 2. In eclipse, conf.get("bsp.master.address") returns null.
>>>
>>>    String mode = conf.get("bsp.master.address");
>>>    LOG.info(">>>>>> " + mode);
>>>
>>> It means that the config files are not loaded. If we run with advanced
>>> configurations in eclipse, it'll be work but, it's really annoying.
>>>
>>> I think, we can set the basic configurations for JUnit tests by
>>> hard-coding in MiniClusterTest constructor.
>>>
>>> On Thu, Apr 14, 2011 at 8:02 PM, chl501 <chl501@nuk.edu.tw> wrote:
>>> > My question should be `Can't we simply use the default value, which sets
>>> bsp.master.address to localhost, in hama-default.xml'?
>>> >
>>> > Currently when using the default value (localhost), the bspMasterAddr
>>> will has the value referenced to master address/port.
>>> >
>>> > Resetting bsp.master.address to local will cause bspMasterAddr to null.
>>> >
>>> > Why do we need to reset bsp.master.address to local?
>>> >
>>> > Sorry for my poor English.
>>> >
>>> > -----Original message-----
>>> > From:chl501 <chl501@nuk.edu.tw>
>>> > To:hama-dev <hama-dev@incubator.apache.org>
>>> > Date:Thu, 14 Apr 2011 17:45:38 +0800 (CST)
>>> > Subject:Re: Eclipse test failure for HAMA-376
>>> >
>>> > Looks like the problem is because BSPMaster.getAddress(conf) in
>>> GroomServer is trying to create socket again after BSPMaster successfully
>>> occupies the ip/port (GroomServers need to wait until BSPMaster is up).
>>> >
>>> > in the end of NetUtil.createSocketAddr() it shows
>>> >
>>> > public static InetSocketAddress createSocketAddr(String target,
>>> >                                                
  int defaultPort) {
>>> >    ...
>>> >    return new InetSocketAddress(hostname, port);
>>> > }
>>> >
>>> > Should we change not to use bspMasterAddr because GroomServer can
>>> directly obtain bsp.master.address (ip/port) from conf/hama-site.xml?
>>> >
>>> > -----Original message-----
>>> > From:Edward J. Yoon <edwardyoon@apache.org>
>>> > To:hama-dev@incubator.apache.org,chl501@nuk.edu.tw
>>> > Date:Thu, 14 Apr 2011 17:32:01 +0900
>>> > Subject:Re: Eclipse test failure for HAMA-376
>>> >
>>> > Nope, "Class not found error" and "setJar()" problems are unconnected
>>> > things from my report.
>>> >
>>> > I think, the point of miniBSPCluster problem with eclipse and maven
>>> > build, is that the value of "bsp.master.address" is always null:
>>> >
>>> >>11/04/12 20:49:50 INFO bsp.GroomServer: groom start
>>> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >
>>> > Try to set the value of "mode" to "local" as below:
>>> >
>>> >  public GroomServer(Configuration conf) throws IOException {
>>> >    ....
>>> >    mode = "local";
>>> >    ....
>>> >
>>> > 2011/4/14 chl501 <chl501@nuk.edu.tw>:
>>> >> It is the same issue that the jar file is missing. If it is required
to
>>> ask use explicitly setJar, is there any place that we can reference to the
>>> build version? For instance, the BSPJob.setJar() requires to set jar path
>>> e.g. /path/to/eclipse/hama-1090935/build/hama-0.3.0-test.jar where build
>>> version 0.3.0 is changed according to ant's property ${version}. Otherwise,
>>> we may need to parse ant file for explicitly adding this info.
>>> >>
>>> >> 11/04/14 14:50:13 WARN bsp.BSPJobClient: No job jar file set.  User
>>> classes may not be found. See BSPJob#setJar(String) or check Your jar file.
>>> >> 11/04/14 14:50:13 INFO bsp.GroomServer: jobConf.getJar():null
>>> >> 11/04/14 14:50:13 WARN bsp.GroomServer: Error initializing
>>> attempt_201104141450_0001_000001_0:
>>> >> java.lang.IllegalArgumentException: Can not create a Path from a null
>>> string
>>> >>        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
>>> >>        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>>> >>        at
>>> org.apache.hama.bsp.GroomServer.localizeJob(GroomServer.java:519)
>>> >>        at
>>> org.apache.hama.bsp.GroomServer.startNewTask(GroomServer.java:451)
>>> >>        at
>>> org.apache.hama.bsp.GroomServer.access$100(GroomServer.java:70)
>>> >>        at
>>> org.apache.hama.bsp.GroomServer$DispatchTasksHandler.handle(GroomServer.java:144)
>>> >>        at
>>> org.apache.hama.bsp.GroomServer$Instructor.run(GroomServer.java:192)
>>> >>
>>> >>
>>> >>>Basically, your miniBSPCluster doesn't work with eclipse JUnit test
and
>>> maven build as below:
>>> >>
>>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Cleaning up the system directory
>>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: file:/tmp/hadoop/bsp/system
>>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: java.lang.NullPointerException
>>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server Responder: starting
>>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server listener on 40000:
>>> starting
>>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server handler 0 on 40000:
>>> starting
>>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Starting RUNNING
>>> >>>11/04/12 20:49:50 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:50 INFO bsp.GroomServer: groom start
>>> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>11/04/12 20:49:56 INFO hama.MiniBSPCluster: Waitin for GroomServer
up.
>>> >>>^CDestroying 1 processes
>>> >>>Destroying process..
>>> >>>Destroyed 1 processes
>>> >>>Can you find out the reason why?
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> ChiaHung Lin
>>> >> Department of Information Management
>>> >> National University of Kaohsiung
>>> >> Taiwan
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > http://blog.udanax.org
>>> > http://twitter.com/eddieyoon
>>> >
>>> >
>>> > ChiaHung Lin
>>> > Department of Information Management
>>> > National University of Kaohsiung
>>> > Taiwan
>>> >
>>> >
>>> > ChiaHung Lin
>>> > Department of Information Management
>>> > National University of Kaohsiung
>>> > Taiwan
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> http://blog.udanax.org
>>> http://twitter.com/eddieyoon
>>>
>>
>>
>>
>> --
>> Thomas Jungblut
>> Berlin
>>
>> mobile: 0170-3081070
>>
>> business: thomas.jungblut@testberichte.de
>> private: thomas.jungblut@gmail.com
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> http://blog.udanax.org
> http://twitter.com/eddieyoon
>



-- 
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon


ChiaHung Lin
Department of Information Management
National University of Kaohsiung
Taiwan

Mime
View raw message