hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2374) Should not use PrintWriter to write taskjvm.sh
Date Wed, 25 Jul 2012 01:13:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421911#comment-13421911
] 

Todd Lipcon commented on MAPREDUCE-2374:
----------------------------------------

bq. Since we are thinking of removing the -c switch, to avoid potential data loss issues (from
delayed allocation by file systems like ext4) I have made some changes so that the IO buffer
contents of taskjvm.sh file are committed to the underlying storage before shell executor
is 

That shouldn't be necessary -- the exec code is perfectly OK to read out of the OS buffer
cache. Adding an fsync here will just hurt the latency of task launch. Though I agree that
a flush() is necessary to make sure it is pushed out of any Java-side buffers.
                
> Should not use PrintWriter to write taskjvm.sh
> ----------------------------------------------
>
>                 Key: MAPREDUCE-2374
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2374
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.1
>
>         Attachments: failed_taskjvmsh.strace, mapreduce-2374-branch-1.patch, mapreduce-2374-on-20sec.txt,
mapreduce-2374.txt, mapreduce-2374.txt, successfull_taskjvmsh.strace
>
>
> Our use of PrintWriter in TaskController.writeCommand is unsafe, since that class swallows
all IO exceptions. We're not currently checking for errors, which I'm seeing result in occasional
task failures with the message "Text file busy" - assumedly because the close() call is failing
silently for some reason.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message