incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: svn commit: r1001586 - in /incubator/hama/trunk: ./ conf/ src/java/org/apache/hama/bsp/
Date Mon, 27 Sep 2010 11:10:00 GMT
You are right, thanks for pointing me. I'll do so ;)

On Mon, Sep 27, 2010 at 8:02 PM, Filipe David Manana
<fdmanana@apache.org> wrote:
> Hi Edward,
>
> Thanks for committing it.
>
> In the future, it would be nicer to commit only the changes related to
> the respective ticket/patch, to make it easier for others to
> follow/review the development.
>
> In this particular case, you committed as well changes relative to the
> job paths. It would be easier for others if those changes came in a
> separate commit.
>
> cheers
>
> On Mon, Sep 27, 2010 at 5:28 AM,  <edwardyoon@apache.org> wrote:
>> Author: edwardyoon
>> Date: Mon Sep 27 04:28:32 2010
>> New Revision: 1001586
>>
>> URL: http://svn.apache.org/viewvc?rev=1001586&view=rev
>> Log:
>> BSPMaster should use the bsp.master.port config property when creating its InetSocketAddr
instance.
>>
>> Modified:
>>    incubator/hama/trunk/CHANGES.txt
>>    incubator/hama/trunk/conf/hama-default.xml
>>    incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java
>>    incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java
>>    incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java
>>    incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java
>>
>> Modified: incubator/hama/trunk/CHANGES.txt
>> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/CHANGES.txt?rev=1001586&r1=1001585&r2=1001586&view=diff
>> ==============================================================================
>> --- incubator/hama/trunk/CHANGES.txt (original)
>> +++ incubator/hama/trunk/CHANGES.txt Mon Sep 27 04:28:32 2010
>> @@ -46,6 +46,8 @@ Trunk (unreleased changes)
>>
>>   IMPROVEMENTS
>>
>> +    HAMA-287: BSPMaster should use the bsp.master.port config property
>> +                       when creating its InetSocketAddr instance (Filipe
Manana via edwardyoon)
>>     HAMA-283: Removing duplicate code (Filipe Manana via edwardyoon)
>>     HAMA-277: Add default number of bsp task (edwardyoon)
>>     HAMA-273: Implement killJob() method for local job (edwardyoon)
>>
>> Modified: incubator/hama/trunk/conf/hama-default.xml
>> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/conf/hama-default.xml?rev=1001586&r1=1001585&r2=1001586&view=diff
>> ==============================================================================
>> --- incubator/hama/trunk/conf/hama-default.xml (original)
>> +++ incubator/hama/trunk/conf/hama-default.xml Mon Sep 27 04:28:32 2010
>> @@ -26,7 +26,8 @@
>>     <name>bsp.master.address</name>
>>     <value>local</value>
>>     <description>The address of the bsp master server. Either the
>> -    literal string "local" or a host:port for distributed mode
>> +    literal string "local" or a host[:port] (where host is a name or
>> +    IP address) for distributed mode.
>>     </description>
>>   </property>
>>   <property>
>>
>> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java
>> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java?rev=1001586&r1=1001585&r2=1001586&view=diff
>> ==============================================================================
>> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java (original)
>> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java Mon Sep 27
04:28:32 2010
>> @@ -237,6 +237,8 @@ public class BSPJobClient extends Config
>>     Path submitJarFile = new Path(submitJobDir, "job.jar");
>>     Path submitJobFile = new Path(submitJobDir, "job.xml");
>>
>> +    LOG.debug("BSPJobClient.submitJobDir: " + submitJobDir);
>> +
>>     /*
>>      * set this user's id in job configuration, so later job files can be
>>      * accessed using this user's id
>>
>> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java
>> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java?rev=1001586&r1=1001585&r2=1001586&view=diff
>> ==============================================================================
>> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java (original)
>> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java Mon Sep 27 04:28:32
2010
>> @@ -237,16 +237,10 @@ public class BSPMaster implements JobSub
>>   }
>>
>>   public static InetSocketAddress getAddress(Configuration conf) {
>> -    String hamaMasterStr = conf.get("bsp.master.address", "localhost:40000");
>> -    return NetUtils.createSocketAddr(hamaMasterStr);
>> -  }
>> -
>> -  public int getPort() {
>> -    return this.conf.getInt("bsp.master.port", 0);
>> -  }
>> +    String hamaMasterStr = conf.get("bsp.master.address", "localhost");
>> +    int defaultPort = conf.getInt("bsp.master.port", 40000);
>>
>> -  public Configuration getConf() {
>> -    return this.conf;
>> +   return NetUtils.createSocketAddr(hamaMasterStr, defaultPort);
>>   }
>>
>>   private static SimpleDateFormat getDateFormat() {
>>
>> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java
>> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java?rev=1001586&r1=1001585&r2=1001586&view=diff
>> ==============================================================================
>> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java (original)
>> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java Mon Sep 27
04:28:32 2010
>> @@ -391,23 +391,32 @@ public class GroomServer implements Runn
>>       this.task = task;
>>     }
>>
>> +    static final String SUBDIR = "groomServer";
>> +
>>     public void launchTask() {
>>       // until job is completed, don't accept new task
>>       acceptNewTasks = false;
>>
>>       try {
>>         // TODO: need to move this code to TaskRunner
>> -        Path localJobFile = new Path(task.getJobFile().replace(systemFS.getUri().toString(),
""));
>> -        String localJarFile = localJobFile.toString().replace(".xml", ".jar");
>> +        task.getJobFile();
>> +        conf.addResource(task.getJobFile());
>> +        BSPJob defaultJobConf = new BSPJob((HamaConfiguration) conf);
>> +
>> +        Path localJobFile =
>> +          defaultJobConf.getLocalPath(SUBDIR+"/"+task.getTaskID()+"/"+"job.xml");
>> +        Path localJarFile =
>> +          defaultJobConf.getLocalPath(SUBDIR+"/"+task.getTaskID()+"/"+"job.jar");
>> +
>>         LOG.debug("localJobFile: "+ localJobFile);
>>
>>         systemFS.copyToLocalFile(new Path(task.getJobFile()), localJobFile);
>> -        systemFS.copyToLocalFile(new Path(task.getJobFile().replace(".xml",
".jar")), new Path(localJarFile));
>> +        systemFS.copyToLocalFile(new Path(task.getJobFile().replace(".xml",
".jar")), localJarFile);
>>
>>         HamaConfiguration conf = new HamaConfiguration();
>>         conf.addResource(localJobFile);
>>         BSPJob jobConf = new BSPJob(conf, task.getJobID());
>> -        jobConf.setJar(localJarFile);
>> +        jobConf.setJar(localJarFile.toString());
>>
>>         BSP bsp = (BSP) ReflectionUtils.newInstance(jobConf.getBspClass(), conf);
>>         bsp.setPeer(bspPeer);
>>
>> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java
>> URL: http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java?rev=1001586&r1=1001585&r2=1001586&view=diff
>> ==============================================================================
>> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java (original)
>> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java Mon Sep
27 04:28:32 2010
>> @@ -76,6 +76,10 @@ class JobInProgress {
>>         + ".xml");
>>     this.localJarFile = master.getLocalPath(BSPMaster.SUBDIR + "/" + jobId
>>         + ".jar");
>> +
>> +    LOG.debug("JobInProgress.localJobFile: " + this.localJobFile);
>> +    LOG.debug("JobInProgress.localJarFile: " + this.localJarFile);
>> +
>>     Path jobDir = master.getSystemDirectoryForJob(jobId);
>>     FileSystem fs = jobDir.getFileSystem(conf);
>>     jobFile = new Path(jobDir, "job.xml");
>>
>>
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>



-- 
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message