hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3356) SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem
Date Wed, 07 May 2008 11:32:55 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Devaraj Das updated HADOOP-3356:
--------------------------------

    Priority: Minor  (was: Critical)

This part of the code must never be hit under normal circumstances for intermediate merges
(during shuffle). I chatted with Arun offline and he agreed on this. 

> SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem,
at times in the InMemoryFileSystem
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3356
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.16.3
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Minor
>             Fix For: 0.18.0
>
>
> The offending code is:
> {code:title=SequenceFile.java}
>             Path outputFile =  lDirAlloc.getLocalPathForWrite(
>                                                 tmpFilename.toString(),
>                                                 approxOutputSize, conf);
>             LOG.debug("writing intermediate results to " + outputFile);
>             Writer writer = cloneFileAttributes(
>                                                 fs.makeQualified(segmentsToMerge.get(0).segmentPathName),

>                                                 fs.makeQualified(outputFile), null);
> {code}
> *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong
FileSystem is used during intermediate merges.
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message