Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 14 Sep 2012 18:06:09 +1100 (NCT)
From: "Sumadhur Reddy Bolli (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <1163090017.79219.1347606369358.JavaMail.jiratomcat@arcas>
In-Reply-To: <52751755.54490.1340673763749.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HDFS-3566?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sumadhur Reddy Bolli updated HDFS-3566:
---------------------------------------

    Attachment: azurepolicy-branch-1-win.patch

submitted a patch for azure policy in branch-1-win
               =20
> Custom Replication Policy for Azure
> -----------------------------------
>
>                 Key: HDFS-3566
>                 URL: https://issues.apache.org/jira/browse/HDFS-3566
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Sumadhur Reddy Bolli
>            Assignee: Sumadhur Reddy Bolli
>             Fix For: 1-win
>
>         Attachments: azurepolicy-branch-1-win.patch
>
>
> Azure has logical concepts like fault and upgrade domains. Each fault dom=
ain spans multiple upgrade domains and each upgrade domain spans multiple f=
ault domains. Machines are spread typically evenly across both fault and up=
grade domains. Fault domain failures are typically catastrophic/unplanned f=
ailures and data loss possibility is high. An upgrade domain can be taken d=
own by azure for maintenance periodically. Each time an upgrade domain is t=
aken down a small percentage of machines in the upgrade domain(typically 1-=
2%) are replaced due to disk failures, thus losing data. Assuming the defau=
lt replication factor 3, any 3 data nodes going down at the same time would=
 mean potential data loss. So, it is important to have a policy that spread=
s replicas across both fault and upgrade domains to ensure practically no d=
ata loss. The problem here is two dimensional and the default policy in had=
oop is one-dimensional. This policy would spread the datanodes across atlea=
st 2 fault domains and three upgrade domains to prevent data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira