hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6032) Unable to check mapreduce job status if submitted using a non-default namenode
Date Mon, 11 Aug 2014 14:51:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092840#comment-14092840
] 

Daryn Sharp commented on MAPREDUCE-6032:
----------------------------------------

{{Filesystem#makeQualified(Path)}} is intended to fail if the fs is already qualified for
another fs.

{{Path#makeQualified(FileSystem)}}, which is oddly deprecated, is intended to qualify against
the given fs only if the path isn't already qualified.  What you found with it mangling a
localfs authority is a nasty bug!

All said, why even bother explicitly qualifying?  This should work:
{code}
stagingDirPath = new Path(stagingDirStr);
statingDirFS = stagingDirPath.getFileSystem(conf);
{code}

> Unable to check mapreduce job status if submitted using a non-default namenode
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6032
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6032
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.0.5-alpha, 2.1.1-beta, 2.0.6-alpha, 2.2.0, 2.3.0, 2.2.1, 2.4.1
>         Environment: Any
>            Reporter: Benjamin Zhitomirsky
>            Assignee: Benjamin Zhitomirsky
>             Fix For: trunk
>
>         Attachments: MAPREDUCE-6032.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When MRv2 job container runs in a context of non-default file system JobHistoryUtils.java
obtains mapreduce.jobhistory.done-dir and
>  mapreduce.jobhistory.intermediate-done-dir as a non-qualified paths (e.g. /mapred/history).
This path is considered to belong to the current container's context. As result the application
history is being written to another file system and job history server is unable to pick it
up, because it expects it to be found on the default file system. Currently providing fully
qualified path to those parameters is not supported as well, because of a bug in JobHistoryEventHandler.
> After this fix two scenarios will be supported:
> - mapreduce.jobhistory.done-dir and mapreduce.jobhistory.intermediate-done-dir (and the
staging directory BTW) will support a fully qualified path
> - If a non-qualified path is configured then it will always be defaulted to the default
file system (core-site.xml). That's how consistency of history location will be archived
> Implementation notes:
>  - FileSystem#makeQualified throws exception if specified path belongs to another file
system. However FileContext#makeQualified work properly in this case, and this is the meaning
of the fix in JobHistoryEventHandler. I was not ready to change behavior FileSystem#makeQualified
because much more thought is required. I afraid that many users expect such behavior, and
fixing it would break their code.
> - The fix in JobHistoryUtils detects non-default namenode configuration only if it comes
from some "real" configuration: core-default.xml is ignored. This is done primary as a kind
of test hook, because otherwise setting fs.defaultFS value during test executions would be
always recognized by  JobHistoryUtils  as a non-default namenode against 'file:///' specified
in core-default.xml. 
> (Remark. Note that makeQualified doesn't behave properly with file:/// filesystem, for
example:
> new Path("file:///dir/subdir").makeQualified(new URI("hdfs://server:8020"), new Path("/dir"))
> Returns: "file://server:8020/dir/subdir" which doesn't make sense.
> However I don't believe it worth fixing, since nobody really case about local file system
besides tests. My fix just ensures that all tests run smoothly by ignoring core-default.xml
file system in the logic.)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message