hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Isaacson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4178) shell scripts should not close stderr
Date Tue, 13 Nov 2012 22:34:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496622#comment-13496622
] 

Andy Isaacson commented on HDFS-4178:
-------------------------------------

bq. The data corruption occurs when the process opens a new file descriptor for its data channel
and gets file descriptor 2

Yep, exactly.  If an app does {{fopen("datafile", "w+")}} (or the Java equivalent) and is
run under {{someapp 2>&-}} the file {{datafile}} will be opened for write as filedescriptor
2.  If the app then triggers glibc's "bad free() detected" codepath (among many others), the
error message will potentially be written to the current file position in {{datafile}}.

(glibc has a workaround for the most common case, it opens {{/dev/tty}} for the error message
in interactive sessions, but it still does corrupt output files when run in a noninteractive
session with {{cron}}, {{at}}, {{batch}}, or at startup time from {{/etc/init.d}}.)
                
> shell scripts should not close stderr
> -------------------------------------
>
>                 Key: HDFS-4178
>                 URL: https://issues.apache.org/jira/browse/HDFS-4178
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 2.0.2-alpha
>            Reporter: Andy Isaacson
>            Assignee: Andy Isaacson
>         Attachments: hdfs4178.txt
>
>
> The {{start-dfs.sh}} and {{stop-dfs.sh}} scripts close stderr for some subprocesses using
the construct
> bq. {{2>&-}}
> This is dangerous because child processes started up under this scenario will re-use
filedescriptor 2 for opened files.  Since libc and many other codepaths assume that filedescriptor
2 can be written to in error conditions, this can potentially result in data corruption.
> Much better to redirect stderr using the construct {{2>/dev/null}}.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message