hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2374) Should not use PrintWriter to write taskjvm.sh
Date Mon, 23 Jul 2012 21:20:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420970#comment-13420970
] 

Colin Patrick McCabe commented on MAPREDUCE-2374:
-------------------------------------------------

Ah, you're absolutely right.  Without the -c, bash won't execve the file itself.  It will
read the file into an already running interpreter.  So no ETXTBSY.

I'm a little nervous about this proposed "fix" because it doesn't really seem to fix the problem.
 Are we absolutely sure that the file is closed and not still being written to?  If it's not,
we could be getting file descriptor leaks, partially written scripts, and all the usual evils.

I guess Shrinivas did some testing with lsof.  Shrinivas, can you try deliberately keeping
the file open for write and verifying that your lsof test detects it?  Also, it would be interesting
to see if the same problem shows up when using the raw FileChannel API (as opposed to the
FileWriter API).

If this is a real kernel bug, I guess we need to file it upstream on the red hat bug tracker
and/or kernel mailing list.
                
> Should not use PrintWriter to write taskjvm.sh
> ----------------------------------------------
>
>                 Key: MAPREDUCE-2374
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2374
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.1
>
>         Attachments: mapreduce-2374-on-20sec.txt, mapreduce-2374.txt, mapreduce-2374.txt
>
>
> Our use of PrintWriter in TaskController.writeCommand is unsafe, since that class swallows
all IO exceptions. We're not currently checking for errors, which I'm seeing result in occasional
task failures with the message "Text file busy" - assumedly because the close() call is failing
silently for some reason.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message