hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3356) SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
Date Wed, 07 May 2008 11:40:56 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594858#action_12594858
] 

devaraj edited comment on HADOOP-3356 at 5/7/08 4:39 AM:
-------------------------------------------------------------

This part of the code must never be hit under normal circumstances for intermediate merges
(during shuffle). We should only do single-level merges for the intermediate merges. I chatted
with Arun offline and he agreed on this. 
Note that this part works as expected when it is supposed to be executed - for multi-level
merges and that happens only at the end of the shuffle (when the fs is the localfs). 
We probably should fix this for completeness sake but it is definitely not a critical/major
issue.

      was (Author: devaraj):
    This part of the code must never be hit under normal circumstances for intermediate merges
(during shuffle). I chatted with Arun offline and he agreed on this. 
  
> SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem,
at times in the InMemoryFileSystem
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3356
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.16.3
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Minor
>             Fix For: 0.18.0
>
>
> The offending code is:
> {code:title=SequenceFile.java}
>             Path outputFile =  lDirAlloc.getLocalPathForWrite(
>                                                 tmpFilename.toString(),
>                                                 approxOutputSize, conf);
>             LOG.debug("writing intermediate results to " + outputFile);
>             Writer writer = cloneFileAttributes(
>                                                 fs.makeQualified(segmentsToMerge.get(0).segmentPathName),

>                                                 fs.makeQualified(outputFile), null);
> {code}
> *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong
FileSystem is used during intermediate merges.
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message