hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumadhur Reddy Bolli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3566) Custom Replication Policy for Azure
Date Tue, 18 Sep 2012 07:18:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457656#comment-13457656

Sumadhur Reddy Bolli commented on HDFS-3566:

Thanks for the comments Eli. I tried running it and it was getting hung. The traces indicated
the following exception. 

 [exec] * [13/23]   [0/0]     0.111s 0b      hdfs_user_guide.pdf
     [exec] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/fop/messaging/MessageHandler
     [exec] 	at org.apache.cocoon.serialization.FOPSerializer.configure(FOPSerializer.java:122)
     [exec] 	at org.apache.avalon.framework.container.ContainerUtil.configure(ContainerUtil.java:201)
     [exec] 	at org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:289)
     [exec] 	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.newPoolable(InstrumentedResourceLimitingPool.java:655)
     [exec] 	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.get(InstrumentedResourceLimitingPool.java:371)
     [exec] 	at org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
     [exec] 	at org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
     [exec] 	at org.apache.

The command I used was ant test-patch -Dpatch.file=/home/sumab/src/branch-1-win/azurepolicy-branch-1-win.patch
-Dfindbugs.home=/home/sumab/src/tools/findbugs-2.0.1 -Djava5.home=/home/sumab/src/tools/java/jdk1.6.0_32

Could you please suggest how to run it on branch-1-win or what could be wrong with what I

> Custom Replication Policy for Azure
> -----------------------------------
>                 Key: HDFS-3566
>                 URL: https://issues.apache.org/jira/browse/HDFS-3566
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Sumadhur Reddy Bolli
>            Assignee: Sumadhur Reddy Bolli
>             Fix For: 1-win
>         Attachments: AzureBlockPlacementPolicy.pdf, azurepolicy-branch-1-win.patch
> Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple
upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread
typically evenly across both fault and upgrade domains. Fault domain failures are typically
catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be
taken down by azure for maintenance periodically. Each time an upgrade domain is taken down
a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk
failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going
down at the same time would mean potential data loss. So, it is important to have a policy
that spreads replicas across both fault and upgrade domains to ensure practically no data
loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional.
This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains
to prevent data loss.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message