hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7551) LocalDirAllocator should incorporate LocalStorage
Date Thu, 13 Oct 2011 06:41:11 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126382#comment-13126382
] 

Ravi Gummadi commented on HADOOP-7551:
--------------------------------------

>> Therefore if there's a disk failure then every call to get a local path will result
in LocalDirAllocator#confChanged doing a disk check of all the configured local dirs.

LocalDirAllocator#AllocatorPerContext#confChanged is actually updating the savedLocalDirs
everytime conf is changed. So the main concern raised in the above description i.e. every
call to confChanged is not resulting into disk checks (except once per configuration change
i.e. once per new bad disk).
                
> LocalDirAllocator should incorporate LocalStorage
> -------------------------------------------------
>
>                 Key: HADOOP-7551
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7551
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.204.0
>            Reporter: Eli Collins
>
> The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in
MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen
to be good. Therefore if there's a disk failure then *every* call to get a local path will
result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs.
It seems like LocalStorage should be a private class to LocalAllocator so that all users of
LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator
don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles
faulty directories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message