hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4933) MR1 final merge asks for length of file it just wrote before flushing it
Date Fri, 11 Jan 2013 21:22:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551532#comment-13551532
] 

Robert Joseph Evans commented on MAPREDUCE-4933:
------------------------------------------------

OK I have dug into this a bit more and I am not as scared as I was previously.  I could definitely
be wrong so if someone could confirm my analysis on this I would appreciate it.

So there are two places where we could get issues.  The first is because we get the file size
after writing and without closing the file.  This is not a huge issue because that size is
only used to approximate how much space will be needed for the final merge file, which we
just wrote out so who cares.

The second case is a race between writing the merge data out to a file and reading it back
in.  This will only happen on the final merge when the reducer has too much buffered in memory
and we need to spill to disk (we exceeded mapred.job.reduce.input.buffer.percent).  But even
then it becomes a race between java's buffered IO getting pushed out to disk and another part
of the code reading it back in.  I have a hard time believing that we are going to lose that
race ever.  i think we can move the back to major, sorry about scaring everyone.
                
> MR1 final merge asks for length of file it just wrote before flushing it
> ------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4933
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4933
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1, task
>    Affects Versions: 1.1.1
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>            Priority: Blocker
>         Attachments: MAPREDUCE-4933-branch-1.patch
>
>
> createKVIterator in ReduceTask contains the following code:
> {code}
>           try {
>             Merger.writeFile(rIter, writer, reporter, job);
>             addToMapOutputFilesOnDisk(fs.getFileStatus(outputPath));
>           } catch (Exception e) {
>             if (null != outputPath) {
>               fs.delete(outputPath, true);
>             }
>             throw new IOException("Final merge failed", e);
>           } finally {
>             if (null != writer) {
>               writer.close();
>             }
>           }
> {code}
> Merger#writeFile() does not close the file after writing it, so when fs.getFileStatus()
is called on it, it may not return the correct length.  This causes bad accounting further
down the line, which can lead to map output data being lost.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message