incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-Hung Lin <cli...@googlemail.com>
Subject Re: Examples setJar problem
Date Wed, 13 Apr 2011 07:32:01 GMT
I think maybe I find out the reason why the bsp.jar parameter is not
set even when a user creates an instance BSPJob(conf, aClass), which
will set the bsp.jar parameter implicitly.

It looks like the ClassLoader retrieved is different from what is
being set in RunJar.

In hama, when BSPJob calls findContainingJar(),
my_class.getClassLoader() returns `sun.misc.Launcher$AppClassLoader'.
It is different from what is set in RunJar, which should be
java.net.URLClassLoader.

When executing the mapred PiEstimator example,
my_class.getClassLoader() is `java.net.URLClassLoader'. So even it
doesn't search/ append build/hadoop-*-dev-examples.jar, which looks
like being referenced by the jar command parameter (i.e. hadoop jar
<jar>), in the script, hadoop still can successfully submit job
without a problem.

Originally my concerns is if this is a problem in BSPMaster/
GroomServer, it may be better to solve this issue in the earlier
stage. For instance, after refactoring, some execution in the patch
was falsely reorganized; then it's safer not to apply the patch, which
would save other developers'/ users' time.

2011/4/12 Edward J. Yoon <edwardyoon@apache.org>:
> hum, well.
>
> Let's learn from other projects!
>
> On Tue, Apr 12, 2011 at 6:09 PM, Chia-Hung Lin <clin4j@googlemail.com> wrote:
>> $HAMA_HOME can be removed so that we can leave user to decide the
>> location of jar file. Would that be a better option?
>>
>> 2011/4/12 Edward J. Yoon <edwardyoon@apache.org>:
>>>>  JAR_FILE=$1
>>>>  if [ -f $JAR_FILE ] ; then
>>>>    CLASSPATH=${CLASSPATH}:$HAMA_HOME/$JAR_FILE;
>>>>  fi
>>>
>>> User can also use the jar command like this:
>>>
>>> # {$HAMA_HOME}/bin/hama jar /home/edward/my.jar
>>>
>>> On Tue, Apr 12, 2011 at 4:59 PM, Chia-Hung Lin <clin4j@googlemail.com>
wrote:
>>>> How if we add
>>>>
>>>>  JAR_FILE=$1
>>>>  if [ -f $JAR_FILE ] ; then
>>>>    CLASSPATH=${CLASSPATH}:$HAMA_HOME/$JAR_FILE;
>>>>  fi
>>>>
>>>> to the section where  "$COMMAND" = "jar" (roughly at the line 165 in bin/hama)?
>>>>
>>>> Also, in build.xml, appending the following line
>>>>
>>>>               <fileset dir="${build.dir}">
>>>>                        <include name="**/*.jar"/>
>>>>                </fileset>
>>>>
>>>> to the end of section  where path id is `test.classpath'
>>>>
>>>> I give a quick test. SerializePrinting, PiEstimator and the new added
>>>> test case working ok.
>>>>
>>>> 2011/4/12 Edward J. Yoon <edwardyoon@apache.org>:
>>>>> Release tarball contains compiled .jar binaries. So user can download
>>>>> release tarball and run without additional compiling source code. At
>>>>> this point if user builds with command 'ant package', another jar
>>>>> files (including duplicated classes) will be created in 'build'
>>>>> directory.
>>>>>
>>>>> ..
>>>>> /hama-0.3.0.jar
>>>>> /build/hama-0.3.0.jar
>>>>>
>>>>> I think, one of them only added to classpath.
>>>>>
>>>>> On Tue, Apr 12, 2011 at 1:42 PM, Chia-Hung Lin <clin4j@googlemail.com>
wrote:
>>>>>> I download the clean trunk from
>>>>>> https://svn.apache.org/repos/asf/incubator/hama/trunk
>>>>>>
>>>>>> After executing `ant tar,` and the extracted tar files contain the
>>>>>> following content.
>>>>>>
>>>>>> bin  build.xml  CHANGES.txt  conf  DISCLAIMER.txt  docs
>>>>>> hama-0.3.0-examples.jar  hama-0.3.0.jar  hama-0.3.0-test.jar  lib
>>>>>> LICENSE.txt  NOTICE.txt  pom.xml  README.txt  src
>>>>>>
>>>>>> Adding the lines that causes the problem and re-executing `ant tar,`
>>>>>> the result seems the same.
>>>>>>
>>>>>> bin  build.xml  CHANGES.txt  conf  DISCLAIMER.txt  docs
>>>>>> hama-0.3.0-examples.jar  hama-0.3.0.jar  hama-0.3.0-test.jar  lib
>>>>>> LICENSE.txt  NOTICE.txt  pom.xml  README.txt  src
>>>>>>
>>>>>> Both of the tar files contains binary jar. Am I missing any step?
Or
>>>>>> how to reproduce this issue?
>>>>>>
>>>>>> 2011/4/11 Edward J. Yoon <edwardyoon@apache.org>:
>>>>>>> Release tarball contains compiled .jar binaries.
>>>>>>>
>>>>>>> On Mon, Apr 11, 2011 at 4:26 PM, Chia-Hung Lin <clin4j@googlemail.com>
wrote:
>>>>>>>> Is there any reason why this function is removed?
>>>>>>>>
>>>>>>>> 2011/4/11 Edward J. Yoon <edwardyoon@apache.org>:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> Oh.... please re-try after run the following command:
>>>>>>>>>
>>>>>>>>> cp build/*.jar ./
>>>>>>>>>
>>>>>>>>> It seems a problem, after below code was removed from
'bin/hama' script.
>>>>>>>>>
>>>>>>>>> for f in $HAMA_HOME/build/hama-*-dev.jar; do # TODO -
should be
>>>>>>>>> changed before initial release
>>>>>>>>>  CLASSPATH=${CLASSPATH}:$f;
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> Do you have some good idea for this problem?
>>>>>>>>>
>>>>>>>>> On Mon, Apr 11, 2011 at 2:13 PM, Chia-Hung Lin <clin4j@googlemail.com>
wrote:
>>>>>>>>>> With the svn revision 1090935, I notice that it requires
user
>>>>>>>>>> explicitly setJar("/path/to/hama-example.jar") so
that the job
>>>>>>>>>> submission will work. Otherwise the GroomServer will
complain
>>>>>>>>>>
>>>>>>>>>>    org.apache.hama.bsp.GroomServer: Lost connection
to BSP Master
>>>>>>>>>> [server/192.168.1.123:40000].  Retrying...
>>>>>>>>>>    java.lang.NullPointerException ...
>>>>>>>>>>
>>>>>>>>>> The steps applied to execute job submission include
>>>>>>>>>>
>>>>>>>>>> ant clean package
>>>>>>>>>> start-dfs.sh
>>>>>>>>>> start-bsp.sh
>>>>>>>>>> hama jar build/hama-0.3.0-examples.jar test
>>>>>>>>>>
>>>>>>>>>> I remember previously when executing examples e.g.
SerializePrinting,
>>>>>>>>>> setJar() method is not required. Am I missing any
step?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>> --
>>>>>>>>>> ChiaHung Lin
>>>>>>>>>> Department of Information Management
>>>>>>>>>> National University of Kaohsiung
>>>>>>>>>> Taiwan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards, Edward J. Yoon
>>>>>>>>> http://blog.udanax.org
>>>>>>>>> http://twitter.com/eddieyoon
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ChiaHung Lin
>>>>>>>> Department of Information Management
>>>>>>>> National University of Kaohsiung
>>>>>>>> Taiwan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards, Edward J. Yoon
>>>>>>> http://blog.udanax.org
>>>>>>> http://twitter.com/eddieyoon
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ChiaHung Lin
>>>>>> Department of Information Management
>>>>>> National University of Kaohsiung
>>>>>> Taiwan
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> http://blog.udanax.org
>>>>> http://twitter.com/eddieyoon
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> ChiaHung Lin
>>>> Department of Information Management
>>>> National University of Kaohsiung
>>>> Taiwan
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> http://blog.udanax.org
>>> http://twitter.com/eddieyoon
>>>
>>
>>
>>
>> --
>> ChiaHung Lin
>> Department of Information Management
>> National University of Kaohsiung
>> Taiwan
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> http://blog.udanax.org
> http://twitter.com/eddieyoon
>



-- 
ChiaHung Lin
Department of Information Management
National University of Kaohsiung
Taiwan

Mime
View raw message