hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Mon, 21 Mar 2016 20:42:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205079#comment-15205079
] 

Tsz Wo Nicholas Sze commented on HDFS-3702:
-------------------------------------------

> So, the suggestion is changing public signatures to add a new parameter (Or adding a
new override where there are already 6)?

For compatibility reason, we probably have to add a new override.  For better usability, we
may add a Builder.

> For a client to make effective use of disfavoredNodes, they would have to figure the
exact name the NN is using and volunteer it in this disfavoredNodes list? Or could they just
write 'localhost' and let NN figure it out?

We should support 'localhost' in the API.  DFSClient or NN may replace 'localhost' with the
corresponding name.

> Do you foresee any other use for this disfavoredNodes parameter other than for the exclusion
of 'localnode'?

Yes, disfavoredNodes seems useful.  For example, some application may want to distribute its
files uniformly in a cluster.  Then, it could specify the previously allocated DNs as the
disfavoredNodes.

> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, HDFS-3702.002.patch, HDFS-3702.003.patch,
HDFS-3702.004.patch, HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, HDFS-3702.008.patch,
HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message