Return-Path: Delivered-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Received: (qmail 44104 invoked from network); 18 Apr 2011 06:17:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 18 Apr 2011 06:17:47 -0000 Received: (qmail 97694 invoked by uid 500); 18 Apr 2011 06:17:47 -0000 Delivered-To: apmail-incubator-hama-dev-archive@incubator.apache.org Received: (qmail 97654 invoked by uid 500); 18 Apr 2011 06:17:46 -0000 Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-dev@incubator.apache.org Received: (qmail 97646 invoked by uid 99); 18 Apr 2011 06:17:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Apr 2011 06:17:46 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=MIME_QP_LONG_LINE,MISSING_MIMEOLE,SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of chl501@nuk.edu.tw does not designate 140.127.234.171 as permitted sender) Received: from [140.127.234.171] (HELO ironport-spam.nuk.edu.tw) (140.127.234.171) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Apr 2011 06:17:38 +0000 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgsZAP7Wq02Mf+oCgWdsb2JhbACETqASgQwBARYmJQ8BDQEdAbIqgwqDYIhtgSmDTnoEhWCPaA X-IronPort-AV: E=Sophos;i="4.64,231,1301846400"; d="scan'208";a="4017386" Received: from lccmail.nuk.edu.tw ([140.127.234.2]) by ironport-spam.nuk.edu.tw with SMTP; 18 Apr 2011 14:17:13 +0800 Received: By OpenMail Mailer;Mon, 18 Apr 2011 14:17:12 +0800 (CST) From: "chl501" Reply-To: chl501@nuk.edu.tw Subject: Re: Eclipse test failure for HAMA-376 Message-ID: <1303107432.8790.chl501@nuk.edu.tw> In-Reply-To: References: <1302767283.3491.chl501@nuk.edu.tw> <1302774338.17293.chl501@nuk.edu.tw> <1302778971.21992.chl501@nuk.edu.tw> To: "hama-dev" Date: Mon, 18 Apr 2011 14:17:12 +0800 (CST) MIME-Version: 1.0 X-Priority: 1 X-MSMail-Priority: High Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Thanks=20for=20the=20explain.=20 Also,=20a=20separate=20ticket=20is=20created=20for=20setJar=20issue.=20 -----Original=20message----- From:Edward=20J.=20Yoon=20 To:hama-dev@incubator.apache.org Date:Fri,=2015=20Apr=202011=2017:56:47=20+0900 Subject:Re:=20Eclipse=20test=20failure=20for=20HAMA-376 P.S.,=20It=20would=20be=20nice=20if=20we=20can=20separate=20setJar()=20probl= em=20into=20another=20ticket! On=20Fri,=20Apr=2015,=202011=20at=205:38=20PM,=20Edward=20J.=20Yoon=20=20wrote: >=20Again,=20.. > >=201.=20JUnit=20tests=20should=20be=20run=20with=20maven=20build=20and=20ec= lipse.=20*=20eclipse >=20is=20not=20a=20point.=20Please=20see=20my=20comment=20again: > >=20https://issues.apache.org/jira/browse/HAMA-376?focusedCommentId=3D130187= 99&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#c= omment-13018799 > >=202.=20By=20default=20("local"),=20user=20should=20able=20to=20run=20the=20= BSP=20job=20without >=20starting=20any=20daemons.=20It's=20not=20for=20unit=20tests. > >=20=C2=A0-=20"local"=20uses=20LocalBSPJobRunner. >=20=C2=A0-=20"localhost"=20means=20that=20the=20'127.0.0.1'=20server. > >=20In=20hadoop=20case,=20by=20default,=20you=20can=20run=20the=20examples=20= using >=20LocalJobRunner=20without=20starting=20hadoop-deamons. > >=20... >=20edward@edward-VirtualBox:~/workspace/hadoop-0.18$=20ps=20-ef=20|=20grep=20= java >=20edward=20=C2=A0=20=C2=A03137=20=C2=A01881=20=C2=A00=2017:23=20pts/0=20=C2= =A0=20=C2=A000:00:00=20grep=20--color=3Dauto=20java > >=20edward@edward-VirtualBox:~/workspace/hadoop-0.18$=20bin/hadoop=20jar >=20build/hadoop-0.18.4-dev-examples.jar=20pi=201=201 >=20Number=20of=20Maps=20=3D=201=20Samples=20per=20Map=20=3D=201 >=20Wrote=20input=20for=20Map=20#0 >=20Starting=20Job >=2011/04/15=2017:23:28=20INFO=20jvm.JvmMetrics:=20Initializing=20JVM=20Metr= ics=20with >=20processName=3DJobTracker,=20sessionId=3D >=2011/04/15=2017:23:28=20INFO=20mapred.FileInputFormat:=20Total=20input=20p= aths=20to=20process=20:=201 >=2011/04/15=2017:23:28=20INFO=20mapred.FileInputFormat:=20Total=20input=20p= aths=20to=20process=20:=201 >=2011/04/15=2017:23:28=20INFO=20mapred.JobClient:=20Running=20job:=20job_lo= cal_0001 >=2011/04/15=2017:23:28=20INFO=20mapred.FileInputFormat:=20Total=20input=20p= aths=20to=20process=20:=201 >=2011/04/15=2017:23:28=20INFO=20mapred.FileInputFormat:=20Total=20input=20p= aths=20to=20process=20:=201 > > > >=20On=20Fri,=20Apr=2015,=202011=20at=204:50=20PM,=20Thomas=20Jungblut >=20=20wrote: >>=20I=20would=20like=20to=20have=20these=20configurations=20a=20tiny=20bit=20= like=20Hadoop.=20Cause=20it's >>=20proven=20and=20easy. >>=20We=20should=20distinguish=20between=20a=20person=20how=20runs=20eclipse= =20and=20want=20to=20develop >>=20a=20BSP,=20real=20cluster=20with=20n-slaves=20and=20a=20(pseudo-distrib= uted)=20one=20machine >>=20cluster. >> >>=20Most=20likely=20when=20a=20user=20downloads=20a=20hama=20release=20he=20= will=20setup=20a >>=20pseudo-distributed=20environment=20on=20his=20linux=20machine. >>=20This=20is=20just=20a=20shortcut=20version=20of=20a=20cluster=20/=20dist= ributed=20version.=20Let=20him >>=20submit=20some=20examples=20and=20look=20whether=20he=20needs=20it=20in=20= his=20production >>=20environment. >>=20If=20he=20decided=20to=20do=20so,=20he=20maybe=20wants=20to=20setup=20a= =20distributed=20environment. >>=20BUT=20he=20really=20needs=20to=20test=20his=20own=20BSP=20and=20develop= =20it.=20It=20is=20not=20easy=20if >>=20you=20are=20not=20familiar=20with=20the=20framework. >>=20The=20LocalBSPJobRunner=20is=20just=20the=20same=20as=20a=20LocalJobRun= ner=20in=20Hadoop=20(this >>=20was=20added=20for=20testing=20mapreduce=20stuff=20on=20your=20local=20m= achine=20AND=20especially >>=20on=20windows,=20without=20submitting=20anything=20to=20a=20production=20= environment=20or >>=20running=20deamons).=20Just=20for=20stepping=20through=20the=20code=20an= d=20getting=20familiar >>=20with=20the=20framework. >> >>=20Nobody=20will=20have=20a=20conf=20directory=20in=20his=20own=20trunk=20= when=20he=20develops=20for=20a >>=20company=20with=20hama. >>=20Most=20of=20them=20will=20setup=20a=20management=20utitily=20where=20co= nfigurations=20are=20stored >>=20and=20jobs=20are=20submitted,=20from=20time=20to=20time=20with=20a=20sc= heduler. >> >>=20Short=20summary=20of=20my=20opinion >> >>=20=C2=A0=201.=20User=20running=20on=20linux=20with=20hama=20deamons=20sho= uld=20receive=20the=20full >>=20=C2=A0=20cluster=20experience=20(just=20the=20way=20it=20is) >>=20=C2=A0=202.=20User=20running=20on=20eclipse/other=20IDE=20should=20rece= ive=20a=20full=20debug=20version >>=20=C2=A0=20that=20emulates=20a=20cluster=20through=20multithreading=20(lo= caljobrunner) >> >>=20Therefore=20we=20should=20not=20add=20anything=20strange=20to=20bsp.mas= ter.address.=20Just=20let >>=20this=20be=20localhost. >> >>=20greetings=20from=20germany >> >>=202011/4/15=20Edward=20J.=20Yoon=20 >> >>>=20I=20think,=20many=20things=20should=20be=20fixed. >>> >>>=201.=20Basically=20the=20value=20of=20"bsp.master.address"=20is=20setted= =20to >>>=20"localhost"=20In=20hama-default.xml: >>> >>>=20=C2=A0=20=C2=A0bsp.master.address >>>=20=C2=A0=20=C2=A0localhost >>>=20=C2=A0=20=C2=A0The=20address=20of=20the=20bsp=20master=20= server.=20Either=20the >>>=20=C2=A0=20=C2=A0literal=20string=20"local"=20or=20a=20host[:port]=20(wh= ere=20host=20is=20a=20name=20or >>> >>>=20If=20we=20add=20a=20LocalBSPJobRunner=20to=20Hama=200.3,=20I=20think,=20= we=20should=20use >>>=20"local"=20as=20a=20default=20value.=20("local"=20=E2=89=A0=20"localhos= t").=20Description >>>=20strings=20should=20be=20fixed,=20too. >>> >>>=202.=20In=20eclipse,=20conf.get("bsp.master.address")=20returns=20null. >>> >>>=20=C2=A0=20=C2=A0String=20mode=20=3D=20conf.get("bsp.master.address"); >>>=20=C2=A0=20=C2=A0LOG.info(">>>>>>=20"=20+=20mode); >>> >>>=20It=20means=20that=20the=20config=20files=20are=20not=20loaded.=20If=20= we=20run=20with=20advanced >>>=20configurations=20in=20eclipse,=20it'll=20be=20work=20but,=20it's=20rea= lly=20annoying. >>> >>>=20I=20think,=20we=20can=20set=20the=20basic=20configurations=20for=20JUn= it=20tests=20by >>>=20hard-coding=20in=20MiniClusterTest=20constructor. >>> >>>=20On=20Thu,=20Apr=2014,=202011=20at=208:02=20PM,=20chl501=20=20wrote: >>>=20>=20My=20question=20should=20be=20`Can't=20we=20simply=20use=20the=20d= efault=20value,=20which=20sets >>>=20bsp.master.address=20to=20localhost,=20in=20hama-default.xml'? >>>=20> >>>=20>=20Currently=20when=20using=20the=20default=20value=20(localhost),=20= the=20bspMasterAddr >>>=20will=20has=20the=20value=20referenced=20to=20master=20address/port. >>>=20> >>>=20>=20Resetting=20bsp.master.address=20to=20local=20will=20cause=20bspMa= sterAddr=20to=20null. >>>=20> >>>=20>=20Why=20do=20we=20need=20to=20reset=20bsp.master.address=20to=20loca= l? >>>=20> >>>=20>=20Sorry=20for=20my=20poor=20English. >>>=20> >>>=20>=20-----Original=20message----- >>>=20>=20From:chl501=20 >>>=20>=20To:hama-dev=20 >>>=20>=20Date:Thu,=2014=20Apr=202011=2017:45:38=20+0800=20(CST) >>>=20>=20Subject:Re:=20Eclipse=20test=20failure=20for=20HAMA-376 >>>=20> >>>=20>=20Looks=20like=20the=20problem=20is=20because=20BSPMaster.getAddress= (conf)=20in >>>=20GroomServer=20is=20trying=20to=20create=20socket=20again=20after=20BSP= Master=20successfully >>>=20occupies=20the=20ip/port=20(GroomServers=20need=20to=20wait=20until=20= BSPMaster=20is=20up). >>>=20> >>>=20>=20in=20the=20end=20of=20NetUtil.createSocketAddr()=20it=20shows >>>=20> >>>=20>=20public=20static=20InetSocketAddress=20createSocketAddr(String=20ta= rget, >>>=20>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2= =A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20= =C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0= =20int=20defaultPort)=20{ >>>=20>=20=C2=A0=20=C2=A0... >>>=20>=20=C2=A0=20=C2=A0return=20new=20InetSocketAddress(hostname,=20port);= >>>=20>=20} >>>=20> >>>=20>=20Should=20we=20change=20not=20to=20use=20bspMasterAddr=20because=20= GroomServer=20can >>>=20directly=20obtain=20bsp.master.address=20(ip/port)=20from=20conf/hama-= site.xml? >>>=20> >>>=20>=20-----Original=20message----- >>>=20>=20From:Edward=20J.=20Yoon=20 >>>=20>=20To:hama-dev@incubator.apache.org,chl501@nuk.edu.tw >>>=20>=20Date:Thu,=2014=20Apr=202011=2017:32:01=20+0900 >>>=20>=20Subject:Re:=20Eclipse=20test=20failure=20for=20HAMA-376 >>>=20> >>>=20>=20Nope,=20"Class=20not=20found=20error"=20and=20"setJar()"=20problem= s=20are=20unconnected >>>=20>=20things=20from=20my=20report. >>>=20> >>>=20>=20I=20think,=20the=20point=20of=20miniBSPCluster=20problem=20with=20= eclipse=20and=20maven >>>=20>=20build,=20is=20that=20the=20value=20of=20"bsp.master.address"=20is=20= always=20null: >>>=20> >>>=20>>11/04/12=2020:49:50=20INFO=20bsp.GroomServer:=20groom=20start >>>=20>>>11/04/12=2020:49:51=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:52=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:53=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:54=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:55=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20> >>>=20>=20Try=20to=20set=20the=20value=20of=20"mode"=20to=20"local"=20as=20b= elow: >>>=20> >>>=20>=20=C2=A0public=20GroomServer(Configuration=20conf)=20throws=20IOExce= ption=20{ >>>=20>=20=C2=A0=20=C2=A0.... >>>=20>=20=C2=A0=20=C2=A0mode=20=3D=20"local"; >>>=20>=20=C2=A0=20=C2=A0.... >>>=20> >>>=20>=202011/4/14=20chl501=20: >>>=20>>=20It=20is=20the=20same=20issue=20that=20the=20jar=20file=20is=20mis= sing.=20If=20it=20is=20required=20to >>>=20ask=20use=20explicitly=20setJar,=20is=20there=20any=20place=20that=20w= e=20can=20reference=20to=20the >>>=20build=20version?=20For=20instance,=20the=20BSPJob.setJar()=20requires=20= to=20set=20jar=20path >>>=20e.g.=20/path/to/eclipse/hama-1090935/build/hama-0.3.0-test.jar=20where= =20build >>>=20version=200.3.0=20is=20changed=20according=20to=20ant's=20property=20$= {version}.=20Otherwise, >>>=20we=20may=20need=20to=20parse=20ant=20file=20for=20explicitly=20adding=20= this=20info. >>>=20>> >>>=20>>=2011/04/14=2014:50:13=20WARN=20bsp.BSPJobClient:=20No=20job=20jar=20= file=20set.=20=C2=A0User >>>=20classes=20may=20not=20be=20found.=20See=20BSPJob#setJar(String)=20or=20= check=20Your=20jar=20file. >>>=20>>=2011/04/14=2014:50:13=20INFO=20bsp.GroomServer:=20jobConf.getJar():= null >>>=20>>=2011/04/14=2014:50:13=20WARN=20bsp.GroomServer:=20Error=20initializ= ing >>>=20attempt_201104141450_0001_000001_0: >>>=20>>=20java.lang.IllegalArgumentException:=20Can=20not=20create=20a=20Pa= th=20from=20a=20null >>>=20string >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at=20org.apache.hadoop.fs.Path.c= heckPathArg(Path.java:78) >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at=20org.apache.hadoop.fs.Path.<= init>(Path.java:90) >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at >>>=20org.apache.hama.bsp.GroomServer.localizeJob(GroomServer.java:519) >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at >>>=20org.apache.hama.bsp.GroomServer.startNewTask(GroomServer.java:451) >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at >>>=20org.apache.hama.bsp.GroomServer.access$100(GroomServer.java:70) >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at >>>=20org.apache.hama.bsp.GroomServer$DispatchTasksHandler.handle(GroomServe= r.java:144) >>>=20>>=20=C2=A0=20=C2=A0=20=C2=A0=20=C2=A0at >>>=20org.apache.hama.bsp.GroomServer$Instructor.run(GroomServer.java:192) >>>=20>> >>>=20>> >>>=20>>>Basically,=20your=20miniBSPCluster=20doesn't=20work=20with=20eclips= e=20JUnit=20test=20and >>>=20maven=20build=20as=20below: >>>=20>> >>>=20>>>11/04/12=2020:49:49=20INFO=20bsp.BSPMaster:=20Cleaning=20up=20the=20= system=20directory >>>=20>>>11/04/12=2020:49:49=20INFO=20bsp.BSPMaster:=20file:/tmp/hadoop/bsp/= system >>>=20>>>11/04/12=2020:49:49=20INFO=20bsp.BSPMaster:=20java.lang.NullPointer= Exception >>>=20>>>11/04/12=2020:49:49=20INFO=20ipc.Server:=20IPC=20Server=20Responder= :=20starting >>>=20>>>11/04/12=2020:49:49=20INFO=20ipc.Server:=20IPC=20Server=20listener=20= on=2040000: >>>=20starting >>>=20>>>11/04/12=2020:49:49=20INFO=20ipc.Server:=20IPC=20Server=20handler=20= 0=20on=2040000: >>>=20starting >>>=20>>>11/04/12=2020:49:49=20INFO=20bsp.BSPMaster:=20Starting=20RUNNING >>>=20>>>11/04/12=2020:49:50=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:50=20INFO=20bsp.GroomServer:=20groom=20start >>>=20>>>11/04/12=2020:49:51=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:52=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:53=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:54=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:55=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>11/04/12=2020:49:56=20INFO=20hama.MiniBSPCluster:=20Waitin=20for=20= GroomServer=20up. >>>=20>>>^CDestroying=201=20processes >>>=20>>>Destroying=20process.. >>>=20>>>Destroyed=201=20processes >>>=20>>>Can=20you=20find=20out=20the=20reason=20why? >>>=20>> >>>=20>> >>>=20>> >>>=20>> >>>=20>> >>>=20>>=20ChiaHung=20Lin >>>=20>>=20Department=20of=20Information=20Management >>>=20>>=20National=20University=20of=20Kaohsiung >>>=20>>=20Taiwan >>>=20>> >>>=20> >>>=20> >>>=20> >>>=20>=20-- >>>=20>=20Best=20Regards,=20Edward=20J.=20Yoon >>>=20>=20http://blog.udanax.org >>>=20>=20http://twitter.com/eddieyoon >>>=20> >>>=20> >>>=20>=20ChiaHung=20Lin >>>=20>=20Department=20of=20Information=20Management >>>=20>=20National=20University=20of=20Kaohsiung >>>=20>=20Taiwan >>>=20> >>>=20> >>>=20>=20ChiaHung=20Lin >>>=20>=20Department=20of=20Information=20Management >>>=20>=20National=20University=20of=20Kaohsiung >>>=20>=20Taiwan >>>=20> >>> >>> >>> >>>=20-- >>>=20Best=20Regards,=20Edward=20J.=20Yoon >>>=20http://blog.udanax.org >>>=20http://twitter.com/eddieyoon >>> >> >> >> >>=20-- >>=20Thomas=20Jungblut >>=20Berlin >> >>=20mobile:=200170-3081070 >> >>=20business:=20thomas.jungblut@testberichte.de >>=20private:=20thomas.jungblut@gmail.com >> > > > >=20-- >=20Best=20Regards,=20Edward=20J.=20Yoon >=20http://blog.udanax.org >=20http://twitter.com/eddieyoon > --=20 Best=20Regards,=20Edward=20J.=20Yoon http://blog.udanax.org http://twitter.com/eddieyoon ChiaHung=20Lin Department=20of=20Information=20Management National=20University=20of=20Kaohsiung Taiwan