hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nijel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9355) Support colocation in HDFS.
Date Wed, 25 Nov 2015 05:56:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026254#comment-15026254

nijel commented on HDFS-9355:

Idea looks good
But co-location is a very broad topic. So i suggest to focus on favored nodes optimization
as part of this JIRA as you mentioned.

One option is to give a client API to get the DNs based on storage policy.

> Support colocation in HDFS.
> ---------------------------
>                 Key: HDFS-9355
>                 URL: https://issues.apache.org/jira/browse/HDFS-9355
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client
>            Reporter: Surendra Singh Lilhore
>            Assignee: Surendra Singh Lilhore
> Through this feature client can give suggestion to HDFS to write his all the blocks on
same set of datanodes. Currently this we can achieve through HDFS-2576. HDFS-2576 give option
to hint namenode about favored nodes, but in heterogeneous cluster this will not work out.
Support client wants to write his data in directory which have COLD policy, but he don't know
which DN have ARCHIVE storage, So he will not able to give favoredNodes list. 
> *Implementation*
> Colocation can enable by setting "dfs.colocation.enable" true in client configuration.
If colocation is enable and  favoredNodes list is empty then {{DataStreamer}} will set first
set of datanodes as favoredNodes which is chosen for first block and subsequent block will
use the same datanodes for write. Before closing file client can get the favoredNodes list
and same he can use for writing new file.

This message was sent by Atlassian JIRA

View raw message