hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba Borthakur" <dhr...@yahoo-inc.com>
Subject RE: DFS Block Allocation
Date Thu, 20 Dec 2007 17:37:55 GMT
Hi Jeff,

Did you run the file-upload command on the master node itself? The DFS
client attempts to store one replica of the data on the node on which
the DFSClient is running.

To get a uniform distribution, it would be good if you upload your data
from multiple nodes in your cluster. 

Thanks,
dhruba


-----Original Message-----
From: Jeff Eastman [mailto:jeastman@collab.net] 
Sent: Thursday, December 20, 2007 7:15 AM
To: hadoop-user@lucene.apache.org
Subject: DFS Block Allocation

I've brought up a small cluster and uploaded some large files. The
master node is cu027 and it seems to be getting an unfair percentage of
the blocks allocated to it, especially compared to cu171 which has the
same size disk. Can somebody shed some light on the reasons for this?

 

Jeff

 

 

Node 

Last Contact 

Admin State 

Size (GB) 

Used (%) 

Blocks 

cu009
<http://cu009.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

0

In Service

69.22

75.07

81 

cu027
<http://cu027.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

1

In Service

291.24

59.74

333 

cu028
<http://cu028.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

1

In Service

71.19

35.34

76 

cu034
<http://cu034.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

1

In Service

71.19

17.79

81 

cu035
<http://cu035.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

0

In Service

71.19

18.49

64 

cu045
<http://cu045.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

1

In Service

71.19

16.68

70 

cu050
<http://cu050.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

2

In Service

71.19

31.99

69 

cu062
<http://cu062.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

2

In Service

71.19

39.64

72 

cu063
<http://cu063.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

2

In Service

71.19

25.03

77 

cu171
<http://cu171.cubit.sp.collab.net:50075/browseDirectory.jsp?namenodeInfo
Port=50070&dir=%2F> 

2

In Service

291.24

13.05

79 

 


Mime
View raw message