hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Mon, 23 Jul 2012 22:59:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421009#comment-13421009
] 

nkeywal commented on HDFS-3702:
-------------------------------

It would work. But we're adding unnecessary local workload, and (the main issue today) we
need on failure to manage the fact that this node is probably dead and should be used only
if the two (or three) others are not available. I see it as: when we write we write at the
wrong place; when we read we need to take into account that we wrote at the wrong place. But
if we write at the right place at the beginning then there is nothing to do in the reads.
I though about marking these blocks as corrupted, but it's really extreme, as these blocks
could be the only ones left in some cases...
                
> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>    Affects Versions: 1.0.3, 2.0.0-alpha
>            Reporter: nkeywal
>            Priority: Minor
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message