hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-8772) RawLocalFileStatus shells out for permission info
Date Wed, 05 Sep 2012 20:53:08 GMT
Daryn Sharp created HADOOP-8772:

             Summary: RawLocalFileStatus shells out for permission info
                 Key: HADOOP-8772
                 URL: https://issues.apache.org/jira/browse/HADOOP-8772
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
            Reporter: Daryn Sharp
            Priority: Critical

{{RawLocalFileStatus}} shells out to run "ls" to get permissions info.  This very inefficient.

More importantly, mixing mulithreading and forking is risky.  Some version of glibc in RHEL
will deadlock things such as "__GI__IO_list_lock" and "malloc_atfork".  All this unnecessary
shelling out to access the local filesystem greatly increases the risk of deadlock.

Namely, the NM's user localizer is seen to jam more frequently than the TT & NM.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message