hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John George (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2081) trying to read configuration in hdfs write path might result in deadlock
Date Sat, 18 Jun 2011 03:13:47 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

John George updated HDFS-2081:
------------------------------

    Attachment: HDFS-2081.patch

Thanks for the pointer Tod. My approach to this is slightly different. 

Attaching a patch without any unit tests for prelim review. I know this patch works because
of the yarn test I ran. I do want to write a unit test for this, so that we know if it breaks
in the future. I dont have a clear cut idea as to how to write one.


> trying to read configuration in hdfs write path might result in deadlock
> ------------------------------------------------------------------------
>
>                 Key: HDFS-2081
>                 URL: https://issues.apache.org/jira/browse/HDFS-2081
>             Project: Hadoop HDFS
>          Issue Type: Bug
>         Environment: MR-279 branch. This is specific to yarn since yarn tries to write
the configuration file to hdfs while holding the lock of the configuration file.
>            Reporter: John George
>            Assignee: John George
>         Attachments: HDFS-2081.patch
>
>
> While trying to run yarn with 1024 block size, it looks as though the dfsclients are
hung waiting to write to DN. The reason is because yarn golds the configuration file lock
and tries to write to hdfs which in turn tries to read from the configuration which requires
the same lock.
> The temporary fix is to make sure that dfsclient threads dont read the config in its
write path. There might be better long term solutions like read/write locks and this needs
to be investigated in another JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message