hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adaryl \"Bob\" Wakefield, MBA" <adaryl.wakefi...@hotmail.com>
Subject Re: hadoop not using whole disk for HDFS
Date Wed, 04 Nov 2015 22:07:25 GMT
/hadoop/hdfs/data

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: P lva 
Sent: Wednesday, November 04, 2015 3:41 PM
To: user@hadoop.apache.org 
Subject: Re: hadoop not using whole disk for HDFS

What does your dfs.datanode.data.dir point to ?



On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA <adaryl.wakefield@hotmail.com>
wrote:

        Filesystem Size Used Avail Use% Mounted on 
        /dev/mapper/centos-root 50G 12G 39G 23% / 
        devtmpfs 16G 0 16G 0% /dev 
        tmpfs 16G 0 16G 0% /dev/shm 
        tmpfs 16G 1.4G 15G 9% /run 
        tmpfs 16G 0 16G 0% /sys/fs/cgroup 
        /dev/sda2 494M 123M 372M 25% /boot 
        /dev/mapper/centos-home 2.7T 33M 2.7T 1% /home 


  That’s from one datanode. The second one is nearly identical. I discovered that 50GB is
actually a default. That seems really weird. Disk space is cheap. Why would you not just use
most of the disk and why is it so hard to reset the default?

  Adaryl "Bob" Wakefield, MBA
  Principal
  Mass Street Analytics, LLC
  913.938.6685
  www.linkedin.com/in/bobwakefieldmba
  Twitter: @BobLovesData

  From: Chris Nauroth 
  Sent: Wednesday, November 04, 2015 12:16 PM
  To: user@hadoop.apache.org 
  Subject: Re: hadoop not using whole disk for HDFS

  How are those drives partitioned?  Is it possible that the directories pointed to by the
dfs.datanode.data.dir property in hdfs-site.xml reside on partitions that are sized to only
100 GB?  Running commands like df would be a good way to check this at the OS level, independently
of Hadoop.

  --Chris Nauroth

  From: MBA <adaryl.wakefield@hotmail.com>
  Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
  Date: Tuesday, November 3, 2015 at 11:16 AM
  To: "user@hadoop.apache.org" <user@hadoop.apache.org>
  Subject: Re: hadoop not using whole disk for HDFS


  Yeah. It has the current value of 1073741824 which is like 1.07 gig.

  B.
  From: Chris Nauroth 
  Sent: Tuesday, November 03, 2015 11:57 AM
  To: user@hadoop.apache.org 
  Subject: Re: hadoop not using whole disk for HDFS

  Hi Bob,

  Does the hdfs-site.xml configuration file contain the property dfs.datanode.du.reserved?
 If this is defined, then the DataNode intentionally will not use this space for storage of
replicas.

  <property>
    <name>dfs.datanode.du.reserved</name>
    <value>0</value>
    <description>Reserved space in bytes per volume. Always leave this much space free
for non dfs use.
    </description>
  </property>

  --Chris Nauroth

  From: MBA <adaryl.wakefield@hotmail.com>
  Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
  Date: Tuesday, November 3, 2015 at 10:51 AM
  To: "user@hadoop.apache.org" <user@hadoop.apache.org>
  Subject: hadoop not using whole disk for HDFS


  I’ve got the Hortonworks distro running on a three node cluster. For some reason the disk
available for HDFS is MUCH less than the total disk space. Both of my data nodes have 3TB
hard drives. Only 100GB of that is being used for HDFS. Is it possible that I have a setting
wrong somewhere? 
  B.

Mime
View raw message