hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ginzburg <ginz...@hotmail.com>
Subject RE: Rack decommission
Date Fri, 11 May 2012 19:30:07 GMT

Thank you,
It has.
I did not expect to see this number >0 since my cluster is configured with rack awareness.

Should I be concerned as to the reliability of the rack awareness? How can I make sure my
rack awareness is configured perfectly, 
beside checking and rechecking it ?
  

Date: Fri, 4 May 2012 09:49:41 -0500
Subject: Re: Rack decommission
From: ravihadoop@gmail.com
To: hdfs-user@hadoop.apache.org

Hi Dave,

I'm not entirely certain, but briefly looking at code:http://search-hadoop.com/c/Hadoop:/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java%7C%7C%252Bebs+%252Bhdfs
Line 2679


            if ((curReplicas == 0) && (num.decommissionedReplicas() > 0)) {
              decommissionOnlyReplicas++;
            }

If the block replica is available ONLY on the node which is being decommissioned, then it
is labeled as "Blocks with No Live Replicas". I would think that this number should go down
after the NN replicates these blocks. Is it?


HTH
Ravi.

On Fri, May 4, 2012 at 5:30 AM, David Ginzburg <ginzman@hotmail.com> wrote:





Thank you,

I am a bit confused about the meaning of the state -"Blocks With No Live Replicas". 
Could not find documentation.
Can you clarify?



Date: Thu, 3 May 2012 10:44:07 -0500

Subject: Re: Rack decommission
From: ravihadoop@gmail.com
To: hdfs-user@hadoop.apache.org


Hi David,

> If a DN is decommissioned and returns, Does the NM update it's block 
meta data. If it does,  How does it decide when to update the meta data 
map?

I presume you meant "the NN update it's block meta data". The answer is yes. Soon as the NN
decides that a node is dead / decommissioned and that blocks have / will fallen below their
replication factor, it will order more replications. This is what is going to happen when
you decommission the nodes.



When the DNs come back, and they report their blocks to the NN, the NN will realise that the
blocks are over-replicated and will order deletions.

> When I view the "dfsnodelist.jsp?whatNodes=

DECOMMISSIONING"  page, there are many slaves with "Blocks With No Live Replicas"
Obviously depends on how loaded your HDFS was. 





On Thu, May 3, 2012 at 10:04 AM, David Ginzburg <ginzman@hotmail.com> wrote:






Hi,
I have began decommissioning the servers  in the rack.
When I view the "dfsnodelist.jsp?whatNodes=DECOMMISSIONING"  page, there are many slaves with
"Blocks With No Live Replicas"
Is it possible , even if I have rack awareness configured ? 





From: ginzman@hotmail.com
To: hdfs-user@hadoop.apache.org; common-user@hadoop.apache.org


Subject: Rack decommission
Date: Wed, 2 May 2012 11:49:41 +0000





Hi,

I have a cluster with 4 racks.
I am scheduled to replace one of them, with slave servers included.
The cluster is configured with rack awareness and replication factor is 3.
It is my understanding that on rack aware clusters, block placement policy ensures blocks
are at least on two racks.


I am concerned about decommission of whole rack.

I considered decommission each node in the rack separately and let it balance automatically
 , but I am afraid it will take a lot of time.

If a DN is decommissioned and returns, Does the NM update it's block meta data. If it does,
 How does it decide when to update the meta data map?



Is there a suggested way to decommission a whole rack , I may have missed?







 		 	   		   		 	   		  

 		 	   		  

 		 	   		  
Mime
View raw message