Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 592F3189FA for ; Wed, 4 Nov 2015 22:27:30 +0000 (UTC) Received: (qmail 86969 invoked by uid 500); 4 Nov 2015 22:27:26 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 86840 invoked by uid 500); 4 Nov 2015 22:27:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 86830 invoked by uid 99); 4 Nov 2015 22:27:25 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2015 22:27:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5BD3F1A2A13 for ; Wed, 4 Nov 2015 22:27:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.97 X-Spam-Level: ** X-Spam-Status: No, score=2.97 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id uTdkOKKIJkbv for ; Wed, 4 Nov 2015 22:27:20 +0000 (UTC) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [58.251.152.64]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id D305942B91 for ; Wed, 4 Nov 2015 22:27:07 +0000 (UTC) Received: from 172.24.1.47 (EHLO szxeml434-hub.china.huawei.com) ([172.24.1.47]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id CYM68641; Thu, 05 Nov 2015 06:26:29 +0800 (CST) Received: from SZXEML505-MBX.china.huawei.com ([169.254.1.20]) by szxeml434-hub.china.huawei.com ([10.82.67.225]) with mapi id 14.03.0235.001; Thu, 5 Nov 2015 06:26:25 +0800 From: "Naganarasimha G R (Naga)" To: "user@hadoop.apache.org" Subject: RE: hadoop not using whole disk for HDFS Thread-Topic: hadoop not using whole disk for HDFS Thread-Index: AQHRFyzpUeymbsqHkkirYtogbq54oZ6L1zwAgAAHUACAAIhz1v//f+SAgACJH7Y= Date: Wed, 4 Nov 2015 22:26:25 +0000 Message-ID: References: <516863ED4DFA3149A9F907F1395E72527ADCBC@blreml508-mbx> , , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.45.28.61] Content-Type: multipart/alternative; boundary="_000_AD354F56741A1B47882A625909A59C692BE3C87ESZXEML505MBXchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.563A8616.0096,ss=1,re=0.000,fgs=0, ip=169.254.1.20, so=2013-06-18 04:22:30, dmn=2011-05-27 18:58:46 X-Mirapoint-Loop-Id: 1ed1988cbb3c340a2dc17dc93399cc6d --_000_AD354F56741A1B47882A625909A59C692BE3C87ESZXEML505MBXchi_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Better would be to stop the daemons and copy the data from /hadoop/hdfs/dat= a to /home/hdfs/data , reconfigure dfs.datanode.data.dir to /home/hdfs/data= and then start the daemons. If the data is comparitively less ! Ensure you have the backup if have any critical data ! Regards, + Naga ________________________________ From: Adaryl "Bob" Wakefield, MBA [adaryl.wakefield@hotmail.com] Sent: Thursday, November 05, 2015 03:40 To: user@hadoop.apache.org Subject: Re: hadoop not using whole disk for HDFS So like I can just create a new folder in the home directory like: home/hdfs/data and then set dfs.datanode.data.dir to: /hadoop/hdfs/data,home/hdfs/data Restart the node and that should do it correct? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.linkedin.com/in/bobwakefieldmba Twitter: @BobLovesData From: Naganarasimha G R (Naga) Sent: Wednesday, November 04, 2015 3:59 PM To: user@hadoop.apache.org Subject: RE: hadoop not using whole disk for HDFS Hi Bob, Seems like you have configured to disk dir to be other than an folder in /h= ome, if so try creating another folder and add to "dfs.datanode.data.dir" s= eperated by comma instead of trying to reset the default. And its also advised not to use the root partition "/" to be configured for= HDFS data dir, if the Dir usage hits the maximum then OS might fail to fun= ction properly. Regards, + Naga ________________________________ From: P lva [ruvikal@gmail.com] Sent: Thursday, November 05, 2015 03:11 To: user@hadoop.apache.org Subject: Re: hadoop not using whole disk for HDFS What does your dfs.datanode.data.dir point to ? On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA > wrote: Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G 12G 39G 23% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 1.4G 15G 9% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda2 494M 123M 372M 25% /boot /dev/mapper/centos-home 2.7T 33M 2.7T 1% /home That=92s from one datanode. The second one is nearly identical. I discovere= d that 50GB is actually a default. That seems really weird. Disk space is c= heap. Why would you not just use most of the disk and why is it so hard to = reset the default? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.linkedin.com/in/bobwakefieldmba Twitter: @BobLovesData From: Chris Nauroth Sent: Wednesday, November 04, 2015 12:16 PM To: user@hadoop.apache.org Subject: Re: hadoop not using whole disk for HDFS How are those drives partitioned? Is it possible that the directories poin= ted to by the dfs.datanode.data.dir property in hdfs-site.xml reside on par= titions that are sized to only 100 GB? Running commands like df would be a= good way to check this at the OS level, independently of Hadoop. --Chris Nauroth From: MBA > Reply-To: "user@hadoop.apache.org" > Date: Tuesday, November 3, 2015 at 11:16 AM To: "user@hadoop.apache.org" > Subject: Re: hadoop not using whole disk for HDFS Yeah. It has the current value of 1073741824 which is like 1.07 gig. B. From: Chris Nauroth Sent: Tuesday, November 03, 2015 11:57 AM To: user@hadoop.apache.org Subject: Re: hadoop not using whole disk for HDFS Hi Bob, Does the hdfs-site.xml configuration file contain the property dfs.datanode= .du.reserved? If this is defined, then the DataNode intentionally will not= use this space for storage of replicas. dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always leave this much s= pace free for non dfs use. --Chris Nauroth From: MBA > Reply-To: "user@hadoop.apache.org" > Date: Tuesday, November 3, 2015 at 10:51 AM To: "user@hadoop.apache.org" > Subject: hadoop not using whole disk for HDFS I=92ve got the Hortonworks distro running on a three node cluster. For some= reason the disk available for HDFS is MUCH less than the total disk space.= Both of my data nodes have 3TB hard drives. Only 100GB of that is being us= ed for HDFS. Is it possible that I have a setting wrong somewhere? B. --_000_AD354F56741A1B47882A625909A59C692BE3C87ESZXEML505MBXchi_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Better would be to stop the daemons and copy the data from /hadoop/h= dfs/data to /home/hdfs/data , reconfigure dfs.datanode.data.dir to /home/hdfs/data and then start the daemons. If the data is compar= itively less !

Ensure you have the backup if have any critical data !

 

Regards,

+ Naga

From: Adaryl "Bob" Wakefield, MB= A [adaryl.wakefield@hotmail.com]
Sent: Thursday, November 05, 2015 03:40
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

So like I can just create a new folder in the home directory like:
home/hdfs/data
and then set dfs.datanode.data.dir to:
/hadoop/hdfs/data,home/hdfs/data
 
Restart the node and that should do it correct?
 
Adar= yl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData
 
Sent: Wednesday, November 04, 2015 3:59 PM
Subject: RE: hadoop not using whole disk for HDFS
 

Hi Bob,

 

Seems like you have configured to disk dir to be other than an folder in= /home, if so try creating another folder and add to "dfs.datanode.data.dir" seperated by comma inste= ad of trying to reset the default.

And its also advised not to use the root partition "/" to be c= onfigured for HDFS data dir, if the Dir usage hits the maximum then OS migh= t fail to function properly.

 

Regards,

+ Naga

From: P lva [ruvikal@gmail.com]
Sent: Thursday, November 05, 2015 03:11
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

What does your dfs.datanode.data.dir point to ?
 
 
On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob= " Wakefield, MBA <adaryl.wakefield@hotmail.com> wrote:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 12G 39G 23% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.4G 15G 9% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda2 494M 123M 372M 25% /boot
/dev/mapper/centos-home 2.7T 33M 2.7T 1% /home
 
That=92s from one datanode. The second one is nearly identical. I disc= overed that 50GB is actually a default. That seems really weird. Disk space= is cheap. Why would you not just use most of the disk and why is it so har= d to reset the default?
 
Adar= yl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
9= 13.938.6685
ww= w.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData
 
Sent: Wednesday, November 04, 2015 12:16 PM
Subject: Re: hadoop not using whole disk for HDFS
 
How are those drives partitioned?  Is it possible that the direct= ories pointed to by the dfs.datanode.data.dir property in hdfs-site.xml res= ide on partitions that are sized to only 100 GB?  Running commands lik= e df would be a good way to check this at the OS level, independently of Hadoop.
 
--Chris Nauroth<= /font>
 
From: MBA <adaryl.wakefield@hotmail.com<= /a>>
Reply-To: "user@hadoop.apache.org" = <user@hadoop= .apache.org>
Date: Tuesday, November 3, 2015 at= 11:16 AM
To: "user@hadoop.apache.org" <user@hadoop.apach= e.org>
Subject: Re: hadoop not using whol= e disk for HDFS
 
Yeah. It has the current value of 1073741824 which is like 1.07 gig.
 
B.
Sent: Tuesday, November 03, 2015 11:57 AM
Subject: Re: hadoop not using whole disk for HDFS
 
Hi Bob,
 
Does the hdfs-site.xml configuration file contain the property dfs.dat= anode.du.reserved?  If this is defined, then the DataNode intentionall= y will not use this space for storage of replicas.
 
<property>
  <name>dfs.datanode.du.reserved</name>
  <value>0</value>
  <description>Reserved space in bytes per volume. Always l= eave this much space free for non dfs use.
  </description>
</property>
 
--Chris Nauroth<= /font>
 
From: MBA <adaryl.wakefield@hotmail.com<= /a>>
Reply-To: "user@hadoop.apache.org" = <user@hadoop= .apache.org>
Date: Tuesday, November 3, 2015 at= 10:51 AM
To: "user@hadoop.apache.org" <user@hadoop.apach= e.org>
Subject: hadoop not using whole di= sk for HDFS
 
I=92ve got the Hortonworks distro running on a three node cluster. For some= reason the disk available for HDFS is MUCH less than the total disk space.= Both of my data nodes have 3TB hard drives. Only 100GB of that is being us= ed for HDFS. Is it possible that I have a setting wrong somewhere?
 
B.
 
--_000_AD354F56741A1B47882A625909A59C692BE3C87ESZXEML505MBXchi_--