hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Max Lapan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-9014) Block placement policy with respect to DN free space
Date Thu, 03 Sep 2015 18:50:47 GMT

     [ https://issues.apache.org/jira/browse/HDFS-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Max Lapan updated HDFS-9014:
----------------------------
    Status: Open  (was: Patch Available)

> Block placement policy with respect to DN free space
> ----------------------------------------------------
>
>                 Key: HDFS-9014
>                 URL: https://issues.apache.org/jira/browse/HDFS-9014
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Max Lapan
>         Attachments: ReplicationPolicyFreeSpace.patch
>
>
> Default block allocation policy (also known as 'replication policy') implemented in NN
is random selection from suitable candidates (rack-local or 'other rack'). This is ok when
all DNs in a cluster has nearly equal amount of storage, but leads to problems when some DNs
are significantly larger than other. In that situation, when NN places new blocks in random
fashion, extra space becomes almost unusable and, in extreme case can lead to 100% usage of
all other 'small' DNs with almost empty 'large', which can lead to various HDFS and MR problems.

> Situation when we have datanodes of different sizes is quite real in large, long-lived
systems when different generations of machines are put in a single cluster.
> To overcome this, I implemented a different block allocation policy which places blocks
with respect to free space available on a DN. Please, consider it for inclusion in hdfs codebase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message