hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9184) Some reducers failing to write final output file to s3.
Date Fri, 01 Feb 2013 18:24:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568935#comment-13568935
] 

Steve Loughran commented on HADOOP-9184:
----------------------------------------

The patch won't apply as it's in the branch 1 layout and build, trunk is split up and Mavenized.
If you could do it for there and try and reduce the odd spurious change, that would be good.

I see three possible causes
# you've hit eventual consistency at play in S3. 
 * Which S3 location are you working with? US-default?
 * Have a go on US-west and see if replicates there.
# There's just a race condition between {{isFile()}} (which calls {{getFileStatus()}} and
the {{getFileStatus()}} in the  following condition. I'd switch to a single {{getFileStatus()}}
up front to eliminate that race, and eliminate the cost of another round trip (which is more
expensive to remote filesystems)
# Atomicity: {{mv}} and {{rm -rf}} aren't atomic on blobstores. We know about this and do
need to come up with a better solution at the end of MR jobs for filesystems that declare
they are blobstores.

Steve
                
> Some reducers failing to write final output file to s3.
> -------------------------------------------------------
>
>                 Key: HADOOP-9184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9184
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Jeremy Karn
>         Attachments: example.pig, HADOOP-9184-branch-0.20.patch, hadoop-9184.patch, task_log.txt
>
>
> We had a Hadoop job that was running 100 reducers with most of the reducers expected
to write out an empty file. When the final output was to an S3 bucket we were finding that
sometimes we were missing a final part file.  This was happening approximately 1 job in 3
(so approximately 1 reducer out of 300 was failing to output the data properly). I've attached
the pig script we were using to reproduce the bug.
> After an in depth look and instrumenting the code we traced the problem to moveTaskOutputs
in FileOutputCommitter.  
> The code there looked like:
> {code}
>     if (fs.isFile(taskOutput)) {
> 	… do stuff …       
>     } else if(fs.getFileStatus(taskOutput).isDir()) {
> 	… do stuff … 
>     }
> {code}
> And what we saw happening is that for the problem jobs neither path was being exercised.
 I've attached the task log of our instrumented code.  In this version we added an else statement
and printed out the line "THIS SEEMS LIKE WE SHOULD NEVER GET HERE …".
> The root cause of this seems to be an eventual consistency issue with S3.  You can see
in the log that the first time moveTaskOutputs is called it finds that the taskOutput is a
directory.  It goes into the isDir() branch and successfully retrieves the list of files in
that directory from S3 (in this case just one file).  This triggers a recursive call to moveTaskOutputs
for the file found in the directory.  But in this pass through moveTaskOutput the temporary
output file can't be found resulting in both branches of the above if statement being skipped
and the temporary file never being moved to the final output location.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message