hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7973) DistributedFileSystem close has severe consequences
Date Fri, 20 Jan 2012 20:42:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190080#comment-13190080

Daryn Sharp commented on HADOOP-7973:

bq.  I guess you would need minidfs cluster instead of running miniMR on top of local filesystem?

I wasn't aware the MR minicluster used local fs, I'll check that.  The beauty is if I've missed
something, the tests will immediately fail.

bq. Also, for the actual change, is this sufficient to avoid the close failure? I thought
that FileSystem throwing the actual error is not from the MapTask directly but the one opened
inside TextInputFormat->LineRecordReader

The recorder reader is opened by {{MapTask#getSplitDetails}} in the framework.  It will now
use a distinct fs object separate from any in the user task code, just like it (accidentally)
used to do.  This is the stream that we are seeing in exceptions.

bq. Does this apply to output stream as well?  If it really does, then something needs to
be changed on the reducer side as well?

Yes, the collector gets a unique fs.  The reducer writes the the raw local filesystem.
> DistributedFileSystem close has severe consequences
> ---------------------------------------------------
>                 Key: HADOOP-7973
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7973
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: HADOOP-7973-2.patch, HADOOP-7973-3.patch, HADOOP-7973-4.patch, HADOOP-7973.patch
> The way {{FileSystem#close}} works is very problematic.  Since the {{FileSystems}} are
cached, any {{close}} by any caller will cause problems for every other reference to it. 
Will add more detail in the comments.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message