hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5183) Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view in choosing DN storage for replica.
Date Wed, 11 Sep 2013 03:37:52 GMT
Junping Du created HDFS-5183:
--------------------------------

             Summary: Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to
have a global view in choosing DN storage for replica.
                 Key: HDFS-5183
                 URL: https://issues.apache.org/jira/browse/HDFS-5183
             Project: Hadoop HDFS
          Issue Type: Sub-task
          Components: datanode, namenode, performance
    Affects Versions: Heterogeneous Storage (HDFS-2832)
            Reporter: Junping Du


Per discussion in HDFS-5157, There are two different ways to handle BlockPlacementPolicy and
ReplicaChoosingPolicy in case of multiple storage types:
 1. Client specifies the required storage type when calling addBlock(..) to NN. BlockPlacementPolicy
in NN chooses a set of datanodes accounting for the storage type. Then, client passes the
required storage type to the datanode set and each datanode chooses a particular storage using
a VolumeChoosingPolicy.
 2. Same as before, client specifies the required storage type when calling addBlock(..) to
NN. Now, BlockPlacementPolicy in NN chooses a set of storages (instead of datanodes). Then,
client writes to the corresponding storages. VolumeChoosingPolicy is no longer needed and
it should be removed.
We think #2 is more powerful as it will bring global view to volume choosing or bring storage
status into consideration in replica choosing, so we propose to combine two polices together.
One concern here is it may increase the load of NameNode as previously volume choosing is
decided by DN. We may verify it later (that's why I put performance in component).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message