hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leon Mergen" <l...@solatis.com>
Subject Small patch to improve mapred.pipes
Date Sat, 09 Aug 2008 19:08:35 GMT
Hello,

Working with the .mapred.pipes class a bit today, I found one bug and one
possible interface improvement:

- Application.java in its constructor assumes that
DistributedCache.getLocalCacheFiles
() always returns non-zero, and returns an array with at least one element
-- appropriate checks and exceptions should be thrown here.
- Submitter.java provides a "submitJob ()" method, yet it acts like it's a
runJob method (and also invokes mapred.JobClient.runClient ()) -- it should
provide two interface methods, one runJob () and one submitJob (), who act
just like the JobClient counterparts.

Here is the small patch that implements both changes, based on the 0.17.1
release source -- just in case anyone cares for this minor improvement.

diff -r ./src/java/org/apache/hadoop/mapred/pipes/Application.java
/usr/local/hadoop/src/java/org/apache/hadoop/mapred/pipes/Application.java
33a34
> import org.apache.hadoop.fs.Path;
79c80,86
<     String executable =
DistributedCache.getLocalCacheFiles(conf)[0].toString();
---
>
>     Path[] executablePath = DistributedCache.getLocalCacheFiles (conf);
>     if (executablePath == null || executablePath.length == 0) {
>             throw new IOException ("Unable to retrieve path to
executable");
>     }
>
>     String executable = executablePath[0].toString();
diff -r ./src/java/org/apache/hadoop/mapred/pipes/Submitter.java
/usr/local/hadoop/src/java/org/apache/hadoop/mapred/pipes/Submitter.java
213c213
<   public static RunningJob submitJob(JobConf conf) throws IOException {
---
>    private static void prepareJob(JobConf conf) throws IOException {
215a216
>
227a229
>
232a235
>
244a248
>
252a257
>
261d265
<     return JobClient.runJob(conf);
263a268,278
>   public static RunningJob submitJob (JobConf conf, JobClient jc) throws
IOException {
>     prepareJob (conf);
>     return jc.submitJob (conf);
>   }
>
>   public static RunningJob runJob (JobConf conf) throws IOException {
>     prepareJob (conf);
>     return JobClient.runJob (conf);
>   }
>
>
322a338
>
357a374
>
417c434
<       submitJob(conf);
---
>       runJob(conf);
diff -r ./src/test/org/apache/hadoop/mapred/pipes/TestPipes.java
/usr/local/hadoop/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java
152c152
<       RunningJob result = Submitter.submitJob(job);
---
>       RunningJob result = Submitter.runJob(job);




-- 
Leon Mergen
http://www.solatis.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message