Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Date: Tue, 26 Jun 2012 01:22:42 +0000 (UTC)
From: "Sumadhur Reddy Bolli (JIRA)" <jira@apache.org>
To: hdfs-dev@hadoop.apache.org
Message-ID: <1560046158.54489.1340673763724.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Created] (HDFS-3566) Custom Replication Policy for Azure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Sumadhur Reddy Bolli created HDFS-3566:
------------------------------------------

             Summary: Custom Replication Policy for Azure
                 Key: HDFS-3566
                 URL: https://issues.apache.org/jira/browse/HDFS-3566
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: name-node
            Reporter: Sumadhur Reddy Bolli


Azure has logical concepts like fault and upgrade domains. Each fault domai=
n spans multiple upgrade domains and each upgrade domain spans multiple fau=
lt domains. Machines are spread typically evenly across both fault and upgr=
ade domains. Fault domain failures are typically catastrophic/unplanned fai=
lures and data loss possibility is high. An upgrade domain can be taken dow=
n by azure for maintenance periodically. Each time an upgrade domain is tak=
en down a small percentage of machines in the upgrade domain(typically 1-2%=
) are replaced due to disk failures, thus losing data. Assuming the default=
 replication factor 3, any 3 data nodes going down at the same time would m=
ean potential data loss. So, it is important to have a policy that spreads =
replicas across both fault and upgrade domains to ensure practically no dat=
a loss. The problem here is two dimensional and the default policy in hadoo=
p is one-dimensional. This policy would spread the datanodes across atleast=
 2 fault domains and three upgrade domains to prevent data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira