hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: hadoop not using whole disk for HDFS
Date Wed, 04 Nov 2015 18:16:03 GMT
How are those drives partitioned?  Is it possible that the directories pointed to by the dfs.datanode.data.dir
property in hdfs-site.xml reside on partitions that are sized to only 100 GB?  Running commands
like df would be a good way to check this at the OS level, independently of Hadoop.

--Chris Nauroth

From: MBA <adaryl.wakefield@hotmail.com<mailto:adaryl.wakefield@hotmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Tuesday, November 3, 2015 at 11:16 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: hadoop not using whole disk for HDFS

Yeah. It has the current value of 1073741824 which is like 1.07 gig.

B.
From: Chris Nauroth<mailto:cnauroth@hortonworks.com>
Sent: Tuesday, November 03, 2015 11:57 AM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: hadoop not using whole disk for HDFS

Hi Bob,

Does the hdfs-site.xml configuration file contain the property dfs.datanode.du.reserved? 
If this is defined, then the DataNode intentionally will not use this space for storage of
replicas.

<property>
  <name>dfs.datanode.du.reserved</name>
  <value>0</value>
  <description>Reserved space in bytes per volume. Always leave this much space free
for non dfs use.
  </description>
</property>

--Chris Nauroth

From: MBA <adaryl.wakefield@hotmail.com<mailto:adaryl.wakefield@hotmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Tuesday, November 3, 2015 at 10:51 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: hadoop not using whole disk for HDFS

I've got the Hortonworks distro running on a three node cluster. For some reason the disk
available for HDFS is MUCH less than the total disk space. Both of my data nodes have 3TB
hard drives. Only 100GB of that is being used for HDFS. Is it possible that I have a setting
wrong somewhere?

B.

Mime
View raw message