hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doris Gu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-10604) What about this?Group DNs and add DN groups--named region to HDFS model , use this region to instead of single DN when saving files.
Date Mon, 11 Jul 2016 07:57:10 GMT

     [ https://issues.apache.org/jira/browse/HDFS-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Doris Gu updated HDFS-10604:
----------------------------
    Description: 
The biggest difference this feature will bring is *making blocks belong to the same file to
save in the same region(DN group).*
So the process will be:
1.Config DN groups, for example
bq.Region1:dn1,dn2,dn3
bq.Region2:dn4,dn5,dn6
bq.Region3:dn7,dn8,dn9,dn10

2.Client uploads a file, first analyze whether this file has any existed blocks:
bq.i)Yes:assign new blocks to the DN group where the existed blocks belong to.
bq.ii)No:assign new blocks to a DN group which is chosen by some certain policy to avoid imbalance.

3.Other related processes,including append,balancer etc. also need to modify as well.   

The benefit we wish is when some DNs are down at the same time, the number of affected files(miss
all replicas) is small.
But we are wondering if this is worth doing or not, or if there are problems we haven't noticed.

  was:
The biggest difference this feature will bring is *strong* making blocks belong to the same
file to save in the same region(DN group).*strong*
So the process will be:
1.Config DN groups, for example
bq.Region1:dn1,dn2,dn3
bq.Region2:dn4,dn5,dn6
bq.Region3:dn7,dn8,dn9,dn10

2.Client uploads a file, first analyze whether this file has any existed blocks:
bq.i)Yes:assign new blocks to the DN group where the existed blocks belong to.
bq.ii)No:assign new blocks to a DN group which is chosen by some certain policy to avoid imbalance.

3.Other related processes,including append,balancer etc. also need to modify as well.   

The benefit we wish is when some DNs are down at the same time, the number of affected files(miss
all replicas) is small.
But we are wondering if this is worth doing or not, or if there are problems we haven't noticed.


> What about this?Group DNs and add DN groups--named region to HDFS model , use this region
to instead of single DN when saving files.
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10604
>                 URL: https://issues.apache.org/jira/browse/HDFS-10604
>             Project: Hadoop HDFS
>          Issue Type: Wish
>            Reporter: Doris Gu
>
> The biggest difference this feature will bring is *making blocks belong to the same file
to save in the same region(DN group).*
> So the process will be:
> 1.Config DN groups, for example
> bq.Region1:dn1,dn2,dn3
> bq.Region2:dn4,dn5,dn6
> bq.Region3:dn7,dn8,dn9,dn10
> 2.Client uploads a file, first analyze whether this file has any existed blocks:
> bq.i)Yes:assign new blocks to the DN group where the existed blocks belong to.
> bq.ii)No:assign new blocks to a DN group which is chosen by some certain policy to avoid
imbalance.
> 3.Other related processes,including append,balancer etc. also need to modify as well.
  
> The benefit we wish is when some DNs are down at the same time, the number of affected
files(miss all replicas) is small.
> But we are wondering if this is worth doing or not, or if there are problems we haven't
noticed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message