hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-5183) Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view in choosing DN storage for replica.
Date Wed, 21 Jan 2015 20:04:35 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arpit Agarwal resolved HDFS-5183.
---------------------------------
      Resolution: Implemented
    Hadoop Flags:   (was: Incompatible change)

Resolving this as Implemented.

As part of HDFS-6584 the first of your two approaches was chosen.
bq. 1. Client specifies the required storage type when calling addBlock(..) to NN. BlockPlacementPolicy
in NN chooses a set of datanodes accounting for the storage type. Then, client passes the
required storage type to the datanode set and each datanode chooses a particular storage using
a VolumeChoosingPolicy.


> Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view
in choosing DN storage for replica.
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-5183
>                 URL: https://issues.apache.org/jira/browse/HDFS-5183
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode, performance
>    Affects Versions: Heterogeneous Storage (HDFS-2832)
>            Reporter: Junping Du
>
> Per discussion in HDFS-5157, There are two different ways to handle BlockPlacementPolicy
and ReplicaChoosingPolicy in case of multiple storage types:
>  1. Client specifies the required storage type when calling addBlock(..) to NN. BlockPlacementPolicy
in NN chooses a set of datanodes accounting for the storage type. Then, client passes the
required storage type to the datanode set and each datanode chooses a particular storage using
a VolumeChoosingPolicy.
>  2. Same as before, client specifies the required storage type when calling addBlock(..)
to NN. Now, BlockPlacementPolicy in NN chooses a set of storages (instead of datanodes). Then,
client writes to the corresponding storages. VolumeChoosingPolicy is no longer needed and
it should be removed.
> We think #2 is more powerful as it will bring global view to volume choosing or bring
storage status into consideration in replica choosing, so we propose to combine two polices
together.
> One concern here is it may increase the load of NameNode as previously volume choosing
is decided by DN. We may verify it later (that's why I put performance in component).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message