hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xi Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
Date Tue, 21 May 2013 21:56:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663457#comment-13663457

Xi Fang commented on MAPREDUCE-5224:

Thanks Chuan and Ivan.
1. For Chuan's comment: I added an assert to check this dir exists indeed. This also addresses
Ivan's 10th comment.
2. For Ivan's comment: 
a. For comments #1, 3, 5, 7, 9, 10, I just followed the comments.
b. For comment #2: I personally think it would be better if we can throw out an exception
rather than swallowing it and setting it back to default file system. As mentioned by Mostafa
(offline),  for example, if someone configured the system dir as http://www.awesome.com/system,
then with the fallback solution the exception saying "HTTP is not supported" will be swallowed
and we'll set the system directory as just /system in the default file system, which doesn't
seem like good behavior. We may want someone explicitly know/handle this at the moment  this
c. For comment #4: I renamed the JobTracker#fs to defaultFs and still keep it just for possible
future use/reference of this variable. 
d. For comment #6, 8: I put the initialization of MiniDFSCluster, MiniMRCluster in the test
case and let setUp() just construct a configuration. In this way, we don't have to throw IOException
in setUp() and test case would fail if my code changes are not applied.

> JobTracker should allow the system directory to be in non-default FS
> --------------------------------------------------------------------
>                 Key: MAPREDUCE-5224
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Xi Fang
>            Assignee: Xi Fang
>            Priority: Minor
>             Fix For: 1-win
>         Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch, MAPREDUCE-5224.patch
>  JobTracker today expects the system directory to be in the default file system
>         if (fs == null) {
>           fs = mrOwner.doAs(new PrivilegedExceptionAction<FileSystem>() {
>             public FileSystem run() throws IOException {
>               return FileSystem.get(conf);
>           }});
>         }
> ...
>   public String getSystemDir() {
>     Path sysDir = new Path(conf.get("mapred.system.dir", "/tmp/hadoop/mapred/system"));
>     return fs.makeQualified(sysDir).toString();
>   }
> In Cloud like Azure the default file system is set as ASV (Windows Azure Blob Storage),
but we would still like the system directory to be in DFS. We should change JobTracker to
allow that.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message