Return-Path: Delivered-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Received: (qmail 16950 invoked from network); 27 Sep 2010 11:02:52 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 Sep 2010 11:02:52 -0000 Received: (qmail 85523 invoked by uid 500); 27 Sep 2010 11:02:51 -0000 Delivered-To: apmail-incubator-hama-dev-archive@incubator.apache.org Received: (qmail 85419 invoked by uid 500); 27 Sep 2010 11:02:48 -0000 Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-dev@incubator.apache.org Received: (qmail 85404 invoked by uid 99); 27 Sep 2010 11:02:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Sep 2010 11:02:48 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of fdmanana@gmail.com designates 209.85.215.47 as permitted sender) Received: from [209.85.215.47] (HELO mail-ew0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Sep 2010 11:02:39 +0000 Received: by ewy7 with SMTP id 7so1217872ewy.6 for ; Mon, 27 Sep 2010 04:02:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=HYs735iYOMy4ODRDWtXodPPevBzbXvIWXUHFkiwkJdk=; b=OCjVa7D9e7cxWz+d6LW3li9lvhIogLoRGsQcQzfayWMSNWuvEpmgqJQVBLjjWMUsGE 2sMoWRxo6C26//Eb/qtFdZpxrtNFhg5j2rNDqwi1zJNf0LvKbt0J64oXc9AMgjO9p7cg tyncF4OJM1EK/706R/PfbBJc/+FWdjiR+y+78= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=DUO9MsJ7exfQvYKzMIyzQtsAYCZXpOkJd7fdLp2esWhCZJTSzuYuZYoCjYAuJvRo5T +7Fdjt6PYpGsvqyYMHKmPZinoLAtJopfBpnKBvooXSqAmP3mGrK83g+oSAT92gOJeuku +61uNw/R4ZrQLm2ydfwss+06+aprvlaDQn6U0= MIME-Version: 1.0 Received: by 10.213.28.196 with SMTP id n4mr5728583ebc.48.1285585339585; Mon, 27 Sep 2010 04:02:19 -0700 (PDT) Sender: fdmanana@gmail.com Received: by 10.213.104.131 with HTTP; Mon, 27 Sep 2010 04:02:19 -0700 (PDT) In-Reply-To: <20100927042833.22ED223889D5@eris.apache.org> References: <20100927042833.22ED223889D5@eris.apache.org> Date: Mon, 27 Sep 2010 12:02:19 +0100 X-Google-Sender-Auth: mlA1N9I_XzPRxDDhJ9OimFabii4 Message-ID: Subject: Re: svn commit: r1001586 - in /incubator/hama/trunk: ./ conf/ src/java/org/apache/hama/bsp/ From: Filipe David Manana To: hama-dev@incubator.apache.org Cc: edwardyoon@apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi Edward, Thanks for committing it. In the future, it would be nicer to commit only the changes related to the respective ticket/patch, to make it easier for others to follow/review the development. In this particular case, you committed as well changes relative to the job paths. It would be easier for others if those changes came in a separate commit. cheers On Mon, Sep 27, 2010 at 5:28 AM, wrote: > Author: edwardyoon > Date: Mon Sep 27 04:28:32 2010 > New Revision: 1001586 > > URL: http://svn.apache.org/viewvc?rev=3D1001586&view=3Drev > Log: > BSPMaster should use the bsp.master.port config property when creating it= s InetSocketAddr instance. > > Modified: > =C2=A0 =C2=A0incubator/hama/trunk/CHANGES.txt > =C2=A0 =C2=A0incubator/hama/trunk/conf/hama-default.xml > =C2=A0 =C2=A0incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClie= nt.java > =C2=A0 =C2=A0incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.= java > =C2=A0 =C2=A0incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServe= r.java > =C2=A0 =C2=A0incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgr= ess.java > > Modified: incubator/hama/trunk/CHANGES.txt > URL: http://svn.apache.org/viewvc/incubator/hama/trunk/CHANGES.txt?rev=3D= 1001586&r1=3D1001585&r2=3D1001586&view=3Ddiff > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- incubator/hama/trunk/CHANGES.txt (original) > +++ incubator/hama/trunk/CHANGES.txt Mon Sep 27 04:28:32 2010 > @@ -46,6 +46,8 @@ Trunk (unreleased changes) > > =C2=A0 IMPROVEMENTS > > + =C2=A0 =C2=A0HAMA-287: BSPMaster should use the bsp.master.port config = property > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 when creating its InetSocketAddr instance (Filipe Manana via edwardy= oon) > =C2=A0 =C2=A0 HAMA-283: Removing duplicate code (Filipe Manana via edward= yoon) > =C2=A0 =C2=A0 HAMA-277: Add default number of bsp task (edwardyoon) > =C2=A0 =C2=A0 HAMA-273: Implement killJob() method for local job (edwardy= oon) > > Modified: incubator/hama/trunk/conf/hama-default.xml > URL: http://svn.apache.org/viewvc/incubator/hama/trunk/conf/hama-default.= xml?rev=3D1001586&r1=3D1001585&r2=3D1001586&view=3Ddiff > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- incubator/hama/trunk/conf/hama-default.xml (original) > +++ incubator/hama/trunk/conf/hama-default.xml Mon Sep 27 04:28:32 2010 > @@ -26,7 +26,8 @@ > =C2=A0 =C2=A0 bsp.master.address > =C2=A0 =C2=A0 local > =C2=A0 =C2=A0 The address of the bsp master server. Either t= he > - =C2=A0 =C2=A0literal string "local" or a host:port for distributed mode > + =C2=A0 =C2=A0literal string "local" or a host[:port] (where host is a n= ame or > + =C2=A0 =C2=A0IP address) for distributed mode. > =C2=A0 =C2=A0 > =C2=A0 > =C2=A0 > > Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.= java > URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apach= e/hama/bsp/BSPJobClient.java?rev=3D1001586&r1=3D1001585&r2=3D1001586&view= =3Ddiff > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java (= original) > +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java M= on Sep 27 04:28:32 2010 > @@ -237,6 +237,8 @@ public class BSPJobClient extends Config > =C2=A0 =C2=A0 Path submitJarFile =3D new Path(submitJobDir, "job.jar"); > =C2=A0 =C2=A0 Path submitJobFile =3D new Path(submitJobDir, "job.xml"); > > + =C2=A0 =C2=A0LOG.debug("BSPJobClient.submitJobDir: " + submitJobDir); > + > =C2=A0 =C2=A0 /* > =C2=A0 =C2=A0 =C2=A0* set this user's id in job configuration, so later j= ob files can be > =C2=A0 =C2=A0 =C2=A0* accessed using this user's id > > Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.jav= a > URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apach= e/hama/bsp/BSPMaster.java?rev=3D1001586&r1=3D1001585&r2=3D1001586&view=3Ddi= ff > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java (ori= ginal) > +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java Mon = Sep 27 04:28:32 2010 > @@ -237,16 +237,10 @@ public class BSPMaster implements JobSub > =C2=A0 } > > =C2=A0 public static InetSocketAddress getAddress(Configuration conf) { > - =C2=A0 =C2=A0String hamaMasterStr =3D conf.get("bsp.master.address", "l= ocalhost:40000"); > - =C2=A0 =C2=A0return NetUtils.createSocketAddr(hamaMasterStr); > - =C2=A0} > - > - =C2=A0public int getPort() { > - =C2=A0 =C2=A0return this.conf.getInt("bsp.master.port", 0); > - =C2=A0} > + =C2=A0 =C2=A0String hamaMasterStr =3D conf.get("bsp.master.address", "l= ocalhost"); > + =C2=A0 =C2=A0int defaultPort =3D conf.getInt("bsp.master.port", 40000); > > - =C2=A0public Configuration getConf() { > - =C2=A0 =C2=A0return this.conf; > + =C2=A0 return NetUtils.createSocketAddr(hamaMasterStr, defaultPort); > =C2=A0 } > > =C2=A0 private static SimpleDateFormat getDateFormat() { > > Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.j= ava > URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apach= e/hama/bsp/GroomServer.java?rev=3D1001586&r1=3D1001585&r2=3D1001586&view=3D= diff > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java (o= riginal) > +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java Mo= n Sep 27 04:28:32 2010 > @@ -391,23 +391,32 @@ public class GroomServer implements Runn > =C2=A0 =C2=A0 =C2=A0 this.task =3D task; > =C2=A0 =C2=A0 } > > + =C2=A0 =C2=A0static final String SUBDIR =3D "groomServer"; > + > =C2=A0 =C2=A0 public void launchTask() { > =C2=A0 =C2=A0 =C2=A0 // until job is completed, don't accept new task > =C2=A0 =C2=A0 =C2=A0 acceptNewTasks =3D false; > > =C2=A0 =C2=A0 =C2=A0 try { > =C2=A0 =C2=A0 =C2=A0 =C2=A0 // TODO: need to move this code to TaskRunner > - =C2=A0 =C2=A0 =C2=A0 =C2=A0Path localJobFile =3D new Path(task.getJobFi= le().replace(systemFS.getUri().toString(), "")); > - =C2=A0 =C2=A0 =C2=A0 =C2=A0String localJarFile =3D localJobFile.toStrin= g().replace(".xml", ".jar"); > + =C2=A0 =C2=A0 =C2=A0 =C2=A0task.getJobFile(); > + =C2=A0 =C2=A0 =C2=A0 =C2=A0conf.addResource(task.getJobFile()); > + =C2=A0 =C2=A0 =C2=A0 =C2=A0BSPJob defaultJobConf =3D new BSPJob((HamaCo= nfiguration) conf); > + > + =C2=A0 =C2=A0 =C2=A0 =C2=A0Path localJobFile =3D > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0defaultJobConf.getLocalPath(SUBDIR+"/= "+task.getTaskID()+"/"+"job.xml"); > + =C2=A0 =C2=A0 =C2=A0 =C2=A0Path localJarFile =3D > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0defaultJobConf.getLocalPath(SUBDIR+"/= "+task.getTaskID()+"/"+"job.jar"); > + > =C2=A0 =C2=A0 =C2=A0 =C2=A0 LOG.debug("localJobFile: "+ localJobFile); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 systemFS.copyToLocalFile(new Path(task.getJob= File()), localJobFile); > - =C2=A0 =C2=A0 =C2=A0 =C2=A0systemFS.copyToLocalFile(new Path(task.getJo= bFile().replace(".xml", ".jar")), new Path(localJarFile)); > + =C2=A0 =C2=A0 =C2=A0 =C2=A0systemFS.copyToLocalFile(new Path(task.getJo= bFile().replace(".xml", ".jar")), localJarFile); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 HamaConfiguration conf =3D new HamaConfigurat= ion(); > =C2=A0 =C2=A0 =C2=A0 =C2=A0 conf.addResource(localJobFile); > =C2=A0 =C2=A0 =C2=A0 =C2=A0 BSPJob jobConf =3D new BSPJob(conf, task.getJ= obID()); > - =C2=A0 =C2=A0 =C2=A0 =C2=A0jobConf.setJar(localJarFile); > + =C2=A0 =C2=A0 =C2=A0 =C2=A0jobConf.setJar(localJarFile.toString()); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 BSP bsp =3D (BSP) ReflectionUtils.newInstance= (jobConf.getBspClass(), conf); > =C2=A0 =C2=A0 =C2=A0 =C2=A0 bsp.setPeer(bspPeer); > > Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress= .java > URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apach= e/hama/bsp/JobInProgress.java?rev=3D1001586&r1=3D1001585&r2=3D1001586&view= =3Ddiff > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java = (original) > +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java = Mon Sep 27 04:28:32 2010 > @@ -76,6 +76,10 @@ class JobInProgress { > =C2=A0 =C2=A0 =C2=A0 =C2=A0 + ".xml"); > =C2=A0 =C2=A0 this.localJarFile =3D master.getLocalPath(BSPMaster.SUBDIR = + "/" + jobId > =C2=A0 =C2=A0 =C2=A0 =C2=A0 + ".jar"); > + > + =C2=A0 =C2=A0LOG.debug("JobInProgress.localJobFile: " + this.localJobFi= le); > + =C2=A0 =C2=A0LOG.debug("JobInProgress.localJarFile: " + this.localJarFi= le); > + > =C2=A0 =C2=A0 Path jobDir =3D master.getSystemDirectoryForJob(jobId); > =C2=A0 =C2=A0 FileSystem fs =3D jobDir.getFileSystem(conf); > =C2=A0 =C2=A0 jobFile =3D new Path(jobDir, "job.xml"); > > > --=20 Filipe David Manana, fdmanana@gmail.com, fdmanana@apache.org "Reasonable men adapt themselves to the world. =C2=A0Unreasonable men adapt the world to themselves. =C2=A0That's why all progress depends on unreasonable men."