hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Roelofs (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2020) Use new FileContext APIs for all mapreduce components
Date Thu, 09 Sep 2010 00:51:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907497#action_12907497

Greg Roelofs commented on MAPREDUCE-2020:


 * {{"fs.AbstractFileSystem.file.impl"}} probably should be new {{JobContext.FOO}} style
   ** 3 instances (at least)
 * Merger.java:  {{merge()}} method madness:  how many do we need?  already had 7; now have
14...  where does it end??
 * MultiFileInputFormat.java:  lose {{import org.apache.hadoop.fs.FileSystem}}:  not used;
slows build, adds confusion
   ** probably ditto MultiFileSplit.java and TestMRAsyncDiskService.java
 * MergeManager.java:  _massive_ pile of duplicated constructor and finalMerge() code:  share!
 (may have previously allowed in case of "short-term transition," but Hadoop API transitions
are _not_ short-term => high risk of mismatch-errors in future changes; should share code
where possible even for "temporary" cases, e.g., by calling private helper function from both


 * still adding trailing whitespace (IFile.java, Merger.java)
   ** if necessary, fire up vim on diff and search:  /^+.* $
 * still bad wraps, e.g.:
+    Deserializer<T> deserializer = (Deserializer<T>) factory
+        .getDeserializer(cls);
   or lack of wrap:
+      job.set("fs.AbstractFileSystem.file.impl", "org.apache.hadoop.fs.local.RawLocalFs");
   ** if line starts with a period, almost guaranteed to be wrong
 * avoid superfluous "this." decorations (Merger.java, MergeManager.java):
+      this.mapOutputsCounter = mergedMapOutputsCounter;
+    this.localFC = null;
+    this.rfc = null;

> Use new FileContext APIs for all mapreduce components 
> ------------------------------------------------------
>                 Key: MAPREDUCE-2020
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2020
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Krishna Ramachandran
>            Assignee: Krishna Ramachandran
>         Attachments: mapred-2020-1.patch, mapred-2020-4.patch, mapred-2020-5.patch, mapred-2020.patch
> Migrate mapreduce components to using improved FileContext APIs implemented in
> HADOOP-4952 and 
> HADOOP-6223

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message