hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From P lva <ruvi...@gmail.com>
Subject Re: hadoop not using whole disk for HDFS
Date Wed, 04 Nov 2015 21:41:04 GMT
What does your dfs.datanode.data.dir point to ?


On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA <
adaryl.wakefield@hotmail.com> wrote:

> Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G 12G
> 39G 23% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G
> 1.4G 15G 9% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda2 494M 123M
> 372M 25% /boot /dev/mapper/centos-home 2.7T 33M 2.7T 1% /home
>
> That’s from one datanode. The second one is nearly identical. I discovered
> that 50GB is actually a default. That seems really weird. Disk space is
> cheap. Why would you not just use most of the disk and why is it so hard to
> reset the default?
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
> *From:* Chris Nauroth <cnauroth@hortonworks.com>
> *Sent:* Wednesday, November 04, 2015 12:16 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: hadoop not using whole disk for HDFS
>
> How are those drives partitioned?  Is it possible that the directories
> pointed to by the dfs.datanode.data.dir property in hdfs-site.xml reside on
> partitions that are sized to only 100 GB?  Running commands like df would
> be a good way to check this at the OS level, independently of Hadoop.
>
> --Chris Nauroth
>
> From: MBA <adaryl.wakefield@hotmail.com>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Date: Tuesday, November 3, 2015 at 11:16 AM
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: Re: hadoop not using whole disk for HDFS
>
> Yeah. It has the current value of 1073741824 which is like 1.07 gig.
>
> B.
> *From:* Chris Nauroth <cnauroth@hortonworks.com>
> *Sent:* Tuesday, November 03, 2015 11:57 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: hadoop not using whole disk for HDFS
>
> Hi Bob,
>
> Does the hdfs-site.xml configuration file contain the property
> dfs.datanode.du.reserved?  If this is defined, then the DataNode
> intentionally will not use this space for storage of replicas.
>
> <property>
>   <name>dfs.datanode.du.reserved</name>
>   <value>0</value>
>   <description>Reserved space in bytes per volume. Always leave this much
> space free for non dfs use.
>   </description>
> </property>
>
> --Chris Nauroth
>
> From: MBA <adaryl.wakefield@hotmail.com>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Date: Tuesday, November 3, 2015 at 10:51 AM
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: hadoop not using whole disk for HDFS
>
> I’ve got the Hortonworks distro running on a three node cluster. For some
> reason the disk available for HDFS is MUCH less than the total disk space.
> Both of my data nodes have 3TB hard drives. Only 100GB of that is being
> used for HDFS. Is it possible that I have a setting wrong somewhere?
>
> B.
>

Mime
View raw message