incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: svn commit: r1001586 - in /incubator/hama/trunk: ./ conf/ src/java/org/apache/hama/bsp/
Date Mon, 27 Sep 2010 11:02:19 GMT
Hi Edward,

Thanks for committing it.

In the future, it would be nicer to commit only the changes related to
the respective ticket/patch, to make it easier for others to
follow/review the development.

In this particular case, you committed as well changes relative to the
job paths. It would be easier for others if those changes came in a
separate commit.

cheers

On Mon, Sep 27, 2010 at 5:28 AM,  <edwardyoon@apache.org> wrote:
> Author: edwardyoon
> Date: Mon Sep 27 04:28:32 2010
> New Revision: 1001586
>
> URL: http://svn.apache.org/viewvc?rev=1001586&view=rev
> Log:
> BSPMaster should use the bsp.master.port config property when creating its InetSocketAddr
instance.
>
> Modified:
>    incubator/hama/trunk/CHANGES.txt
>    incubator/hama/trunk/conf/hama-default.xml
>    incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java
>    incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java
>    incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java
>    incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java
>
> Modified: incubator/hama/trunk/CHANGES.txt
> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/CHANGES.txt?rev=1001586&r1=1001585&r2=1001586&view=diff
> ==============================================================================
> --- incubator/hama/trunk/CHANGES.txt (original)
> +++ incubator/hama/trunk/CHANGES.txt Mon Sep 27 04:28:32 2010
> @@ -46,6 +46,8 @@ Trunk (unreleased changes)
>
>   IMPROVEMENTS
>
> +    HAMA-287: BSPMaster should use the bsp.master.port config property
> +                       when creating its InetSocketAddr instance (Filipe
Manana via edwardyoon)
>     HAMA-283: Removing duplicate code (Filipe Manana via edwardyoon)
>     HAMA-277: Add default number of bsp task (edwardyoon)
>     HAMA-273: Implement killJob() method for local job (edwardyoon)
>
> Modified: incubator/hama/trunk/conf/hama-default.xml
> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/conf/hama-default.xml?rev=1001586&r1=1001585&r2=1001586&view=diff
> ==============================================================================
> --- incubator/hama/trunk/conf/hama-default.xml (original)
> +++ incubator/hama/trunk/conf/hama-default.xml Mon Sep 27 04:28:32 2010
> @@ -26,7 +26,8 @@
>     <name>bsp.master.address</name>
>     <value>local</value>
>     <description>The address of the bsp master server. Either the
> -    literal string "local" or a host:port for distributed mode
> +    literal string "local" or a host[:port] (where host is a name or
> +    IP address) for distributed mode.
>     </description>
>   </property>
>   <property>
>
> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java
> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java?rev=1001586&r1=1001585&r2=1001586&view=diff
> ==============================================================================
> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java (original)
> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java Mon Sep 27 04:28:32
2010
> @@ -237,6 +237,8 @@ public class BSPJobClient extends Config
>     Path submitJarFile = new Path(submitJobDir, "job.jar");
>     Path submitJobFile = new Path(submitJobDir, "job.xml");
>
> +    LOG.debug("BSPJobClient.submitJobDir: " + submitJobDir);
> +
>     /*
>      * set this user's id in job configuration, so later job files can be
>      * accessed using this user's id
>
> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java
> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java?rev=1001586&r1=1001585&r2=1001586&view=diff
> ==============================================================================
> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java (original)
> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java Mon Sep 27 04:28:32
2010
> @@ -237,16 +237,10 @@ public class BSPMaster implements JobSub
>   }
>
>   public static InetSocketAddress getAddress(Configuration conf) {
> -    String hamaMasterStr = conf.get("bsp.master.address", "localhost:40000");
> -    return NetUtils.createSocketAddr(hamaMasterStr);
> -  }
> -
> -  public int getPort() {
> -    return this.conf.getInt("bsp.master.port", 0);
> -  }
> +    String hamaMasterStr = conf.get("bsp.master.address", "localhost");
> +    int defaultPort = conf.getInt("bsp.master.port", 40000);
>
> -  public Configuration getConf() {
> -    return this.conf;
> +   return NetUtils.createSocketAddr(hamaMasterStr, defaultPort);
>   }
>
>   private static SimpleDateFormat getDateFormat() {
>
> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java
> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java?rev=1001586&r1=1001585&r2=1001586&view=diff
> ==============================================================================
> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java (original)
> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java Mon Sep 27 04:28:32
2010
> @@ -391,23 +391,32 @@ public class GroomServer implements Runn
>       this.task = task;
>     }
>
> +    static final String SUBDIR = "groomServer";
> +
>     public void launchTask() {
>       // until job is completed, don't accept new task
>       acceptNewTasks = false;
>
>       try {
>         // TODO: need to move this code to TaskRunner
> -        Path localJobFile = new Path(task.getJobFile().replace(systemFS.getUri().toString(),
""));
> -        String localJarFile = localJobFile.toString().replace(".xml", ".jar");
> +        task.getJobFile();
> +        conf.addResource(task.getJobFile());
> +        BSPJob defaultJobConf = new BSPJob((HamaConfiguration) conf);
> +
> +        Path localJobFile =
> +          defaultJobConf.getLocalPath(SUBDIR+"/"+task.getTaskID()+"/"+"job.xml");
> +        Path localJarFile =
> +          defaultJobConf.getLocalPath(SUBDIR+"/"+task.getTaskID()+"/"+"job.jar");
> +
>         LOG.debug("localJobFile: "+ localJobFile);
>
>         systemFS.copyToLocalFile(new Path(task.getJobFile()), localJobFile);
> -        systemFS.copyToLocalFile(new Path(task.getJobFile().replace(".xml", ".jar")),
new Path(localJarFile));
> +        systemFS.copyToLocalFile(new Path(task.getJobFile().replace(".xml", ".jar")),
localJarFile);
>
>         HamaConfiguration conf = new HamaConfiguration();
>         conf.addResource(localJobFile);
>         BSPJob jobConf = new BSPJob(conf, task.getJobID());
> -        jobConf.setJar(localJarFile);
> +        jobConf.setJar(localJarFile.toString());
>
>         BSP bsp = (BSP) ReflectionUtils.newInstance(jobConf.getBspClass(), conf);
>         bsp.setPeer(bspPeer);
>
> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java
> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java?rev=1001586&r1=1001585&r2=1001586&view=diff
> ==============================================================================
> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java (original)
> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java Mon Sep 27 04:28:32
2010
> @@ -76,6 +76,10 @@ class JobInProgress {
>         + ".xml");
>     this.localJarFile = master.getLocalPath(BSPMaster.SUBDIR + "/" + jobId
>         + ".jar");
> +
> +    LOG.debug("JobInProgress.localJobFile: " + this.localJobFile);
> +    LOG.debug("JobInProgress.localJarFile: " + this.localJarFile);
> +
>     Path jobDir = master.getSystemDirectoryForJob(jobId);
>     FileSystem fs = jobDir.getFileSystem(conf);
>     jobFile = new Path(jobDir, "job.xml");
>
>
>



-- 
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
View raw message