hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (Naga)" <garlanaganarasi...@huawei.com>
Subject RE: hadoop not using whole disk for HDFS
Date Fri, 06 Nov 2015 01:50:43 GMT
Hi Bob,



1. I wasn’t able to set the config to /home/hdfs/data. I got an error that told me I’m
not allowed to set that config to the /home directory. So I made it /hdfs/data.

Naga : I am not sure about the HDP Distro but if you make it point to /hdfs/data, still it
will be pointing to the root mount itself i.e.

    /dev/mapper/centos-root     50G     12G     39G     23%     /

Other Alternative is to mount the drive to some other folder other than /home and then try.



2. When I restarted, the space available increased by a whopping 100GB.

Naga : I am particularly not sure how this happened may be you can again recheck if you enter
the command "df -h <path of the NM data dir configured>"  you will find out how much
disk space is available on the related mount for which the path is configured.



Regards,

+ Naga







________________________________

From: Adaryl "Bob" Wakefield, MBA [adaryl.wakefield@hotmail.com]
Sent: Friday, November 06, 2015 06:54
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

Is there a maximum amount of disk space that HDFS will use? Is 100GB that max? When we’re
supposed to be dealing with “big data” why is the amount of data to be held on any one
box such a small number when you’ve got terabytes available?

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: Adaryl "Bob" Wakefield, MBA<mailto:adaryl.wakefield@hotmail.com>
Sent: Wednesday, November 04, 2015 4:38 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: hadoop not using whole disk for HDFS

This is an experimental cluster and there isn’t anything I can’t lose. I ran into some
issues. I’m running the Hortonworks distro and am managing things through Ambari.

1. I wasn’t able to set the config to /home/hdfs/data. I got an error that told me I’m
not allowed to set that config to the /home directory. So I made it /hdfs/data.
2. When I restarted, the space available increased by a whopping 100GB.



Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: Naganarasimha G R (Naga)<mailto:garlanaganarasimha@huawei.com>
Sent: Wednesday, November 04, 2015 4:26 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: hadoop not using whole disk for HDFS


Better would be to stop the daemons and copy the data from /hadoop/hdfs/data to /home/hdfs/data
, reconfigure dfs.datanode.data.dir to /home/hdfs/data and then start the daemons. If the
data is comparitively less !

Ensure you have the backup if have any critical data !



Regards,

+ Naga

________________________________
From: Adaryl "Bob" Wakefield, MBA [adaryl.wakefield@hotmail.com]
Sent: Thursday, November 05, 2015 03:40
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

So like I can just create a new folder in the home directory like:
home/hdfs/data
and then set dfs.datanode.data.dir to:
/hadoop/hdfs/data,home/hdfs/data

Restart the node and that should do it correct?

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: Naganarasimha G R (Naga)<mailto:garlanaganarasimha@huawei.com>
Sent: Wednesday, November 04, 2015 3:59 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: hadoop not using whole disk for HDFS


Hi Bob,



Seems like you have configured to disk dir to be other than an folder in /home, if so try
creating another folder and add to "dfs.datanode.data.dir" seperated by comma instead of trying
to reset the default.

And its also advised not to use the root partition "/" to be configured for HDFS data dir,
if the Dir usage hits the maximum then OS might fail to function properly.



Regards,

+ Naga

________________________________
From: P lva [ruvikal@gmail.com]
Sent: Thursday, November 05, 2015 03:11
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

What does your dfs.datanode.data.dir point to ?


On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA <adaryl.wakefield@hotmail.com<mailto:adaryl.wakefield@hotmail.com>>
wrote:
Filesystem      Size    Used    Avail   Use%    Mounted on
/dev/mapper/centos-root 50G     12G     39G     23%     /
devtmpfs        16G     0       16G     0%      /dev
tmpfs   16G     0       16G     0%      /dev/shm
tmpfs   16G     1.4G    15G     9%      /run
tmpfs   16G     0       16G     0%      /sys/fs/cgroup
/dev/sda2       494M    123M    372M    25%     /boot
/dev/mapper/centos-home 2.7T    33M     2.7T    1%      /home

That’s from one datanode. The second one is nearly identical. I discovered that 50GB is
actually a default. That seems really weird. Disk space is cheap. Why would you not just use
most of the disk and why is it so hard to reset the default?

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685<tel:913.938.6685>
www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba>
Twitter: @BobLovesData

From: Chris Nauroth<mailto:cnauroth@hortonworks.com>
Sent: Wednesday, November 04, 2015 12:16 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: hadoop not using whole disk for HDFS

How are those drives partitioned?  Is it possible that the directories pointed to by the dfs.datanode.data.dir
property in hdfs-site.xml reside on partitions that are sized to only 100 GB?  Running commands
like df would be a good way to check this at the OS level, independently of Hadoop.

--Chris Nauroth

From: MBA <adaryl.wakefield@hotmail.com<mailto:adaryl.wakefield@hotmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Tuesday, November 3, 2015 at 11:16 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: hadoop not using whole disk for HDFS

Yeah. It has the current value of 1073741824 which is like 1.07 gig.

B.
From: Chris Nauroth<mailto:cnauroth@hortonworks.com>
Sent: Tuesday, November 03, 2015 11:57 AM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: hadoop not using whole disk for HDFS

Hi Bob,

Does the hdfs-site.xml configuration file contain the property dfs.datanode.du.reserved? 
If this is defined, then the DataNode intentionally will not use this space for storage of
replicas.

<property>
  <name>dfs.datanode.du.reserved</name>
  <value>0</value>
  <description>Reserved space in bytes per volume. Always leave this much space free
for non dfs use.
  </description>
</property>

--Chris Nauroth

From: MBA <adaryl.wakefield@hotmail.com<mailto:adaryl.wakefield@hotmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Tuesday, November 3, 2015 at 10:51 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: hadoop not using whole disk for HDFS

I’ve got the Hortonworks distro running on a three node cluster. For some reason the disk
available for HDFS is MUCH less than the total disk space. Both of my data nodes have 3TB
hard drives. Only 100GB of that is being used for HDFS. Is it possible that I have a setting
wrong somewhere?

B.


Mime
View raw message