hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Namikaze Minato <lloydsen...@gmail.com>
Subject Re: hadoop not using whole disk for HDFS
Date Sun, 08 Nov 2015 00:10:07 GMT
I hope you understand that you sent 5 emails to several hundred (thousand?)
people in the world in 15 minutes... Please think before hitting this
"send" button.

In Unix (AND windows) you can mount a drive into a folder. This means just
that the disk is accessible from that folder, it does not increase the
capacity of / to mount a 2 TB drive in /home. Nor does it use any space on
/ to do so.
Just think that / is one drive, which contains everything EXCEPT /home and
is for example 50GB big and /home is another drive which is 2TB big.

What you need is to make your hadoop understand that it should use /home
(to be precise a folder in /home and not the complete partition) as hdfs
storage space. Now I will let the other people in the thread disscuss with
you about the technicalities of setting that parameter in the right config
file, as I don't have the knowledge about this specific matter.

Regards,
LLoyd

On 8 November 2015 at 00:00, Adaryl "Bob" Wakefield, MBA <
adaryl.wakefield@hotmail.com> wrote:

> No it’s flat out saying that that config cannot be set with anything
> starting with /home.
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
> *From:* Naganarasimha G R (Naga) <garlanaganarasimha@huawei.com>
> *Sent:* Thursday, November 05, 2015 10:58 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: hadoop not using whole disk for HDFS
>
> Hi Bob,
>
> I am suspecting Ambari would not be allowing to create a folder directly
> under */home*, might be it will allow */home/<user_name>/hdfs*, since
> directories under /home is expected to be users home dir.
>
> Regards,
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga) [garlanaganarasimha@huawei.com]
> *Sent:* Friday, November 06, 2015 09:34
> *To:* user@hadoop.apache.org
> *Subject:* RE: hadoop not using whole disk for HDFS
>
> Thanks Brahma, dint realize he might have configured both directories and
> i was assuming bob has configured single new directory "/hdfs/data".
> So virtually its showing additional space,
> *manually try to add a data dir in /home, for your usecase, and restart
> datanodes.*
> Not sure about the impacs in Ambari but worth a try! , more permanent
> solution would be better remount
> Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-home 2.7T
> 33M 2.7T 1% /home
> ------------------------------
> *From:* Brahma Reddy Battula [brahmareddy.battula@huawei.com]
> *Sent:* Friday, November 06, 2015 08:19
> *To:* user@hadoop.apache.org
> *Subject:* RE: hadoop not using whole disk for HDFS
>
>
> For each configured *dfs.datanode.data.dir* , HDFS thinks its in separate
> partiotion and counts the capacity separately. So when another dir is added
> /hdfs/data, HDFS thinks new partition is added, So it increased the
> capacity 50GB per node. i.e. 100GB for 2 Nodes.
>
> Not allowing /home directory to configure for data.dir might be ambari's
> constraint, instead you can *manually try to add a data dir* in /home,
> for your usecase, and restart datanodes.
>
>
>
> Thanks & Regards
>
>  Brahma Reddy Battula
>
>
>
>
> ------------------------------
> *From:* Naganarasimha G R (Naga) [garlanaganarasimha@huawei.com]
> *Sent:* Friday, November 06, 2015 7:20 AM
> *To:* user@hadoop.apache.org
> *Subject:* RE: hadoop not using whole disk for HDFS
>
> Hi Bob,
>
>
>
> *1. I wasn’t able to set the config to /home/hdfs/data. I got an error
> that told me I’m not allowed to set that config to the /home directory. So
> I made it /hdfs/data.*
>
> *Naga : *I am not sure about the HDP Distro but if you make it point to */hdfs/data,
> *still it will be pointing to the root mount itself i.e.
>
> *    /dev/mapper/centos-root* *50G* *12G* *39G* *23%* */*
>
> Other Alternative is to mount the drive to some other folder other than
> /home and then try.
>
>
> *2. When I restarted, the space available increased by a whopping 100GB.*
>
> *Naga : *I am particularly not sure how this happened may be you can
> again recheck if you enter the command *"df -h <path of the NM data dir
> configured>" *you will find out how much disk space is available on the
> related mount for which the path is configured.
>
>
>
> Regards,
>
> + Naga
>
>
>
>
>
>
> ------------------------------
> *From:* Adaryl "Bob" Wakefield, MBA [adaryl.wakefield@hotmail.com]
> *Sent:* Friday, November 06, 2015 06:54
> *To:* user@hadoop.apache.org
> *Subject:* Re: hadoop not using whole disk for HDFS
>
> Is there a maximum amount of disk space that HDFS will use? Is 100GB that
> max? When we’re supposed to be dealing with “big data” why is the amount of
> data to be held on any one box such a small number when you’ve got
> terabytes available?
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
> *From:* Adaryl "Bob" Wakefield, MBA <adaryl.wakefield@hotmail.com>
> *Sent:* Wednesday, November 04, 2015 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: hadoop not using whole disk for HDFS
>
> This is an experimental cluster and there isn’t anything I can’t lose. I
> ran into some issues. I’m running the Hortonworks distro and am managing
> things through Ambari.
>
> 1. I wasn’t able to set the config to /home/hdfs/data. I got an error that
> told me I’m not allowed to set that config to the /home directory. So I
> made it /hdfs/data.
> 2. When I restarted, the space available increased by a whopping 100GB.
>
>
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
> *From:* Naganarasimha G R (Naga) <garlanaganarasimha@huawei.com>
> *Sent:* Wednesday, November 04, 2015 4:26 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: hadoop not using whole disk for HDFS
>
>
> Better would be to stop the daemons and copy the data from */hadoop/hdfs/data
> *to */home/hdfs/data *, reconfigure *dfs.datanode.data.dir* to */home/hdfs/data
> *and then start the daemons. If the data is comparitively less !
>
> Ensure you have the backup if have any critical data !
>
>
>
> Regards,
>
> + Naga
> ------------------------------
> *From:* Adaryl "Bob" Wakefield, MBA [adaryl.wakefield@hotmail.com]
> *Sent:* Thursday, November 05, 2015 03:40
> *To:* user@hadoop.apache.org
> *Subject:* Re: hadoop not using whole disk for HDFS
>
> So like I can just create a new folder in the home directory like:
> home/hdfs/data
> and then set dfs.datanode.data.dir to:
> /hadoop/hdfs/data,home/hdfs/data
>
> Restart the node and that should do it correct?
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
> *From:* Naganarasimha G R (Naga) <garlanaganarasimha@huawei.com>
> *Sent:* Wednesday, November 04, 2015 3:59 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: hadoop not using whole disk for HDFS
>
>
> Hi Bob,
>
>
>
> Seems like you have configured to disk dir to be other than an folder in*
> /home,* if so try creating another folder and add to
> *"dfs.datanode.data.dir"* seperated by comma instead of trying to reset
> the default.
>
> And its also advised not to use the root partition "/" to be configured
> for HDFS data dir, if the Dir usage hits the maximum then OS might fail to
> function properly.
>
>
>
> Regards,
>
> + Naga
> ------------------------------
> *From:* P lva [ruvikal@gmail.com]
> *Sent:* Thursday, November 05, 2015 03:11
> *To:* user@hadoop.apache.org
> *Subject:* Re: hadoop not using whole disk for HDFS
>
> What does your dfs.datanode.data.dir point to ?
>
>
> On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA <
> adaryl.wakefield@hotmail.com> wrote:
>
>> Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G
>> 12G 39G 23% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm
>> tmpfs 16G 1.4G 15G 9% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda2
>> 494M 123M 372M 25% /boot /dev/mapper/centos-home 2.7T 33M 2.7T 1% /home
>>
>> That’s from one datanode. The second one is nearly identical. I
>> discovered that 50GB is actually a default. That seems really weird. Disk
>> space is cheap. Why would you not just use most of the disk and why is it
>> so hard to reset the default?
>>
>> Adaryl "Bob" Wakefield, MBA
>> Principal
>> Mass Street Analytics, LLC
>> 913.938.6685
>> www.linkedin.com/in/bobwakefieldmba
>> Twitter: @BobLovesData
>>
>> *From:* Chris Nauroth <cnauroth@hortonworks.com>
>> *Sent:* Wednesday, November 04, 2015 12:16 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: hadoop not using whole disk for HDFS
>>
>> How are those drives partitioned?  Is it possible that the directories
>> pointed to by the dfs.datanode.data.dir property in hdfs-site.xml reside on
>> partitions that are sized to only 100 GB?  Running commands like df would
>> be a good way to check this at the OS level, independently of Hadoop.
>>
>> --Chris Nauroth
>>
>> From: MBA <adaryl.wakefield@hotmail.com>
>> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Date: Tuesday, November 3, 2015 at 11:16 AM
>> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Subject: Re: hadoop not using whole disk for HDFS
>>
>> Yeah. It has the current value of 1073741824 which is like 1.07 gig.
>>
>> B.
>> *From:* Chris Nauroth <cnauroth@hortonworks.com>
>> *Sent:* Tuesday, November 03, 2015 11:57 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: hadoop not using whole disk for HDFS
>>
>> Hi Bob,
>>
>> Does the hdfs-site.xml configuration file contain the property
>> dfs.datanode.du.reserved?  If this is defined, then the DataNode
>> intentionally will not use this space for storage of replicas.
>>
>> <property>
>>   <name>dfs.datanode.du.reserved</name>
>>   <value>0</value>
>>   <description>Reserved space in bytes per volume. Always leave this much
>> space free for non dfs use.
>>   </description>
>> </property>
>>
>> --Chris Nauroth
>>
>> From: MBA <adaryl.wakefield@hotmail.com>
>> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Date: Tuesday, November 3, 2015 at 10:51 AM
>> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Subject: hadoop not using whole disk for HDFS
>>
>> I’ve got the Hortonworks distro running on a three node cluster. For some
>> reason the disk available for HDFS is MUCH less than the total disk space.
>> Both of my data nodes have 3TB hard drives. Only 100GB of that is being
>> used for HDFS. Is it possible that I have a setting wrong somewhere?
>>
>> B.
>>
>
>

Mime
View raw message