Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Wed, 11 Sep 2013 03:37:52 +0000 (UTC)
From: "Junping Du (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12667973.1378870654493.113867.1378870672365@arcas>
In-Reply-To: <JIRA.12667973.1378870654493@arcas>
References: <JIRA.12667973.1378870654493@arcas>
Subject: [jira] [Created] (HDFS-5183) Combine ReplicaPlacementPolicy with
 VolumeChoosingPolicy together to have a global view in choosing DN storage
 for replica.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Junping Du created HDFS-5183:
--------------------------------

             Summary: Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view in choosing DN storage for replica.
                 Key: HDFS-5183
                 URL: https://issues.apache.org/jira/browse/HDFS-5183
             Project: Hadoop HDFS
          Issue Type: Sub-task
          Components: datanode, namenode, performance
    Affects Versions: Heterogeneous Storage (HDFS-2832)
            Reporter: Junping Du


Per discussion in HDFS-5157, There are two different ways to handle BlockPlacementPolicy and ReplicaChoosingPolicy in case of multiple storage types:
 1. Client specifies the required storage type when calling addBlock(..) to NN. BlockPlacementPolicy in NN chooses a set of datanodes accounting for the storage type. Then, client passes the required storage type to the datanode set and each datanode chooses a particular storage using a VolumeChoosingPolicy.
 2. Same as before, client specifies the required storage type when calling addBlock(..) to NN. Now, BlockPlacementPolicy in NN chooses a set of storages (instead of datanodes). Then, client writes to the corresponding storages. VolumeChoosingPolicy is no longer needed and it should be removed.
We think #2 is more powerful as it will bring global view to volume choosing or bring storage status into consideration in replica choosing, so we propose to combine two polices together.
One concern here is it may increase the load of NameNode as previously volume choosing is decided by DN. We may verify it later (that's why I put performance in component).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira