hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shrinivas Joshi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2374) Should not use PrintWriter to write taskjvm.sh
Date Fri, 08 Jun 2012 20:53:23 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291984#comment-13291984
] 

Shrinivas Joshi commented on MAPREDUCE-2374:
--------------------------------------------

We are noticing these "Text file busy" exceptions on an Hadoop 1.0.3 environment which are
causing task failures with exit status 126. These exceptions are non-deterministic. They occur
when shExec.execute() call is made in org.apache.hadoop.mapred.DefaultTaskController class.
Return string from the call to shExec.getOutput()is empty line. We captured the details of
this exception by logging the stack trace in the catch block.
Here is what the stack trace looks like:
org.apache.hadoop.util.Shell$ExitCodeException: bash: /local4/1_0_3/hadoop-local/ttprivate/taskTracker/root/jobcache/job_201206081243_0001/attempt_201206081243_0001_m_000024_0/taskjvm.sh:
Text file busy

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
        at org.apache.hadoop.util.Shell.run(Shell.java:182)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
        at org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:131)
        at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:498)
        at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:472)

Also, all the log/work files corresponding to the failed tasks get cleaned up and hence it
becomes harder to debug the underlying cause.

Above patch has helped reduce these failures, however, they still occur. I am not sure whether
the "Text file busy" exception is a Hadoop issue or some other issue though. I just thought
sharing this information here would benefit someone.
                
> Should not use PrintWriter to write taskjvm.sh
> ----------------------------------------------
>
>                 Key: MAPREDUCE-2374
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2374
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.1
>
>         Attachments: mapreduce-2374-on-20sec.txt
>
>
> Our use of PrintWriter in TaskController.writeCommand is unsafe, since that class swallows
all IO exceptions. We're not currently checking for errors, which I'm seeing result in occasional
task failures with the message "Text file busy" - assumedly because the close() call is failing
silently for some reason.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message