hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicolas Liochon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Mon, 21 Mar 2016 23:46:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205380#comment-15205380
] 

Nicolas Liochon commented on HDFS-3702:
---------------------------------------

bq. The issue was opened in July 2012 so we not holding our breath
If we're not holding our breath is also because we put a hack in HBase (HBASE-6435). However,
this hack is not perfect and does not help on the write path (we write and flush 3 times while
two would provide the same level of safety), and we still try to do a recoverLease on a dead
node when there is a server crash.
bq. Yeah, vendors could ensure installers set the attribute.
imho, it's not an optional behavior for HBase (compared to favoredNode which was supposed
to be a power-user configuration only): out of the box, HBase WALs should be written to 2
remote nodes by default, and never to the local node. So it would be much better to have the
right behavior without requiring any extra work, scripts to run or code to deploy on the hdfs
namenode (it's too easy to mess things up).

> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, HDFS-3702.002.patch, HDFS-3702.003.patch,
HDFS-3702.004.patch, HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, HDFS-3702.008.patch,
HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message