hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Krishna Ramachandran (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-2020) Use new FileContext APIs for all mapreduce components
Date Wed, 22 Sep 2010 21:56:33 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Krishna Ramachandran updated MAPREDUCE-2020:
--------------------------------------------

    Attachment: mapred-2020-7.patch

Updated patch with more components using FileContext APIs
Changes since last rev (so only need to focus on these)

Updated  src/java/org/apache/hadoop/mapred/JobTracker.java (due to trunk changes)

filecache
---------
 src/java/org/apache/hadoop/mapreduce/filecache/TaskDistributedCache.java
 src/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java
 src/java/org/apache/hadoop/mapreduce/filecache/TrackerDistributedCacheManager.java

Job Client components
--------------------------

 src/java/org/apache/hadoop/mapreduce/Cluster.java
 src/java/org/apache/hadoop/mapreduce/Job.java
 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java

Others
----------
 src/java/org/apache/hadoop/mapred/JobInProgress.java
 src/java/org/apache/hadoop/mapred/LocalJobRunner.java
 src/java/org/apache/hadoop/mapred/lib/FilterOutputFormat.java
 src/java/org/apache/hadoop/mapred/lib/NullOutputFormat.java
 src/java/org/apache/hadoop/mapred/lib/db/DBOutputFormat.java
 src/java/org/apache/hadoop/mapred/FileOutputFormat.java

All public APIs with dependence on  FileSystem are left alone for compatibility (tools like
oozie or pig may use these)

Most private references/APIs are migrated

Though the migration was straight forward some inconsistent usage caused problems

Example,

path.getFileSystem(conf) (no equivalent API and DFS team's recommendation is not to use this)

getting RawFileSystem FileSystem.getRaw() - workaround 
copyTo/FromLocal - use a generalize copy() from FileContext.Util class which requires fully
qualified path
fs.getUri() - not available via context (workaround)
FileSystem.closeAllforUGI() - not available yet

getFileSystemName() - is a public API in JT why this is needed  





> Use new FileContext APIs for all mapreduce components 
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-2020
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2020
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Krishna Ramachandran
>            Assignee: Krishna Ramachandran
>         Attachments: mapred-2020-1.patch, mapred-2020-4.patch, mapred-2020-5.patch, mapred-2020-6.patch,
mapred-2020-7.patch, mapred-2020.patch
>
>
> Migrate mapreduce components to using improved FileContext APIs implemented in
> HADOOP-4952 and 
> HADOOP-6223

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message