hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Mon, 23 Jul 2012 17:10:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420791#comment-13420791
] 

Suresh Srinivas commented on HDFS-3702:
---------------------------------------

bq. One option might be to put in place a block policy that wrote the first replica local
for all files but those that had a WAL-looking file path
A mechanism to choose block placement policy makes sense. Instead of based on file name, choosing
block placement policy during file creation/append would be more generic. We could allow configuration
to include named block placement policies, such as "default", "hbase-wal" etc. This can be
either passed in create method call as an option or through Configuration when FileSystem
instance is created.

Related question, should the block placement chosen persisted per file or only create/append
time. If it is not persisted, during active replication it is possible that replicas end up
in ways where the block placement policy is not satisfied.

stack and nkeywal, when a node dies, there is a correlated failure and replica count goes
down to two. Is this a big problem? HDFS does create an additional replica right?
                
> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>    Affects Versions: 1.0.3, 2.0.0-alpha
>            Reporter: nkeywal
>            Priority: Minor
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message