hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Small patch to improve mapred.pipes
Date Sat, 09 Aug 2008 19:25:13 GMT
Leon,

On Aug 9, 2008, at 12:08 PM, Leon Mergen wrote:

> Hello,
>
> Working with the .mapred.pipes class a bit today, I found one bug  
> and one
> possible interface improvement:
>

Do you mind opening a jira and attaching this patch there?

thanks,
Arun

> - Application.java in its constructor assumes that
> DistributedCache.getLocalCacheFiles
> () always returns non-zero, and returns an array with at least one  
> element
> -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like  
> it's a
> runJob method (and also invokes mapred.JobClient.runClient ()) -- it  
> should
> provide two interface methods, one runJob () and one submitJob (),  
> who act
> just like the JobClient counterparts.
>
> Here is the small patch that implements both changes, based on the  
> 0.17.1
> release source -- just in case anyone cares for this minor  
> improvement.
>
> diff -r ./src/java/org/apache/hadoop/mapred/pipes/Application.java
> /usr/local/hadoop/src/java/org/apache/hadoop/mapred/pipes/ 
> Application.java
> 33a34
>> import org.apache.hadoop.fs.Path;
> 79c80,86
> <     String executable =
> DistributedCache.getLocalCacheFiles(conf)[0].toString();
> ---
>>
>>    Path[] executablePath = DistributedCache.getLocalCacheFiles  
>> (conf);
>>    if (executablePath == null || executablePath.length == 0) {
>>            throw new IOException ("Unable to retrieve path to
> executable");
>>    }
>>
>>    String executable = executablePath[0].toString();
> diff -r ./src/java/org/apache/hadoop/mapred/pipes/Submitter.java
> /usr/local/hadoop/src/java/org/apache/hadoop/mapred/pipes/ 
> Submitter.java
> 213c213
> <   public static RunningJob submitJob(JobConf conf) throws  
> IOException {
> ---
>>   private static void prepareJob(JobConf conf) throws IOException {
> 215a216
>>
> 227a229
>>
> 232a235
>>
> 244a248
>>
> 252a257
>>
> 261d265
> <     return JobClient.runJob(conf);
> 263a268,278
>>  public static RunningJob submitJob (JobConf conf, JobClient jc)  
>> throws
> IOException {
>>    prepareJob (conf);
>>    return jc.submitJob (conf);
>>  }
>>
>>  public static RunningJob runJob (JobConf conf) throws IOException {
>>    prepareJob (conf);
>>    return JobClient.runJob (conf);
>>  }
>>
>>
> 322a338
>>
> 357a374
>>
> 417c434
> <       submitJob(conf);
> ---
>>      runJob(conf);
> diff -r ./src/test/org/apache/hadoop/mapred/pipes/TestPipes.java
> /usr/local/hadoop/src/test/org/apache/hadoop/mapred/pipes/ 
> TestPipes.java
> 152c152
> <       RunningJob result = Submitter.submitJob(job);
> ---
>>      RunningJob result = Submitter.runJob(job);
>
>
>
>
> -- 
> Leon Mergen
> http://www.solatis.com


Mime
View raw message