hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saumitra Shahapure <saumitra.offic...@gmail.com>
Subject Re: HDFS file system size issue
Date Tue, 15 Apr 2014 13:54:16 GMT
Hi Rahman,

These are few lines from hadoop fsck / -blocks -files -locations

/mnt/hadoop/hive/warehouse/user.db/table1/000255_0 44323326 bytes, 1
block(s):  OK
0. blk_-7919979022650423857_446500 len=44323326 repl=3 [ip1:50010,
ip2:50010, ip3:50010]

/mnt/hadoop/hive/warehouse/user.db/table1/000256_0 44566965 bytes, 1
block(s):  OK
0. blk_-5768999994812882540_446288 len=44566965 repl=3 [ip1:50010,
ip2:50010, ip4:50010]


Biswa may have guessed replication factor from fsck summary that I posted
earlier. I am posting it again for today's run:

Status: HEALTHY
 Total size:    58143055251 B
 Total dirs:    307
 Total files:   5093
 Total blocks (validated):      3903 (avg. block size 14897016 B)
 Minimally replicated blocks:   3903 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       92 (2.357161 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     3.1401486
 Corrupt blocks:                0
 Missing replicas:              92 (0.75065273 %)
 Number of data-nodes:          9
 Number of racks:               1
FSCK ended at Tue Apr 15 13:20:25 UTC 2014 in 655 milliseconds


The filesystem under path '/' is HEALTHY

I have not overridden dfs.datanode.du.reserved. It defaults to 0.

$ less $HADOOP_HOME/conf/hdfs-site.xml |grep -A3 'dfs.datanode.du.reserved'
$ less $HADOOP_HOME/src/hdfs/hdfs-default.xml |grep -A3
'dfs.datanode.du.reserved'
  <name>dfs.datanode.du.reserved</name>
  <value>0</value>
  <description>Reserved space in bytes per volume. Always leave this much
space free for non dfs use.
  </description>

Below is du -h on every node. FYI, my dfs.data.dir is /mnt/hadoop/dfs/data
and all hadoop/hive logs are dumped in /mnt/logs in various directories.
All machines have 400GB for /mnt.

$for i in `echo $dfs_slaves`; do  ssh $i 'du -sh /mnt/hadoop; du -sh
/mnt/hadoop/dfs/data; du -sh /mnt/logs;'; done


225G    /mnt/hadoop
224G    /mnt/hadoop/dfs/data
61M     /mnt/logs

281G    /mnt/hadoop
281G    /mnt/hadoop/dfs/data
63M     /mnt/logs

139G    /mnt/hadoop
139G    /mnt/hadoop/dfs/data
68M     /mnt/logs

135G    /mnt/hadoop
134G    /mnt/hadoop/dfs/data
92M     /mnt/logs

165G    /mnt/hadoop
164G    /mnt/hadoop/dfs/data
75M     /mnt/logs

137G    /mnt/hadoop
137G    /mnt/hadoop/dfs/data
95M     /mnt/logs

160G    /mnt/hadoop
160G    /mnt/hadoop/dfs/data
74M     /mnt/logs

180G    /mnt/hadoop
122G    /mnt/hadoop/dfs/data
23M     /mnt/logs

139G    /mnt/hadoop
138G    /mnt/hadoop/dfs/data
76M     /mnt/logs



All these numbers are for today, and may differ bit from yesterday.

Today hadoop dfs -dus is 58GB and namenode is reporting DFS Used as 1.46TB.

Pardon me for making the mail dirty by lot of copy-pastes, hope it's still
readable,

-- Saumitra S. Shahapure


On Tue, Apr 15, 2014 at 2:57 AM, Abdelrahman Shettia <
ashettia@hortonworks.com> wrote:

> Hi Biswa,
>
> Are you sure that the replication factor of the files are three? Please
> run a ‘hadoop fsck / -blocks -files -locations’ and see the replication
> factor for each file.  Also, Post the configuration of <name>dfs.datanode.
> du.reserved</name> and please check the real space presented by a
> DataNode by running ‘du -h’
>
> Thanks,
> Rahman
>
> On Apr 14, 2014, at 2:07 PM, Saumitra <saumitra.official@gmail.com> wrote:
>
> Hello,
>
> Biswanath, looks like we have confusion in calculation, 1TB would be equal
> to 1024GB, not 114GB.
>
>
> Sandeep, I checked log directory size as well. Log directories are hardly
> in few GBs, I have configured log4j properties so that logs won’t be too
> large.
>
> In our slave machines, we have 450GB disk partition for hadoop logs and
> DFS. Over there logs directory is < 10GBs and rest space is occupied by
> DFS. 10GB partition is for /.
>
> Let me quote my confusion point once again:
>
>  Basically I wanted to point out discrepancy in name node status page and hadoop
>>> dfs -dus. In my case, earlier one reports DFS usage as 1TB and later
>>> one reports it to be 35GB. What are the factors that can cause this
>>> difference? And why is just 35GB data causing DFS to hit its limits?
>>>
>>
>
> I am talking about name node status page on 50070 port. Here is the
> screenshot of my name node status page
>
> <Screen Shot 2014-04-15 at 2.07.19 am.png>
>
> As I understand, 'DFS used’ is the space taken by DFS, non-DFS used is
> spaces taken by non-DFS data like logs or other local files from users.
> Namenode shows that DFS used is ~1TB but hadoop dfs -dus shows it to be
> ~38GB.
>
>
>
> On 14-Apr-2014, at 12:33 pm, Sandeep Nemuri <nhsandeep6@gmail.com> wrote:
>
>  Please check your logs directory usage.
>
>
>
> On Mon, Apr 14, 2014 at 12:08 PM, Biswajit Nayak <
> biswajit.nayak@inmobi.com> wrote:
>
>> Whats the replication factor you have? I believe it should be 3. hadoop
>> dus shows that disk usage without replication. While name node ui page
>> gives with replication.
>>
>> 38gb * 3 =114gb ~ 1TB
>>
>> ~Biswa
>> -----oThe important thing is not to stop questioning o-----
>>
>>
>> On Mon, Apr 14, 2014 at 9:38 AM, Saumitra <saumitra.official@gmail.com>wrote:
>>
>>> Hi Biswajeet,
>>>
>>> Non-dfs usage is ~100GB over the cluster. But still the number are
>>> nowhere near 1TB.
>>>
>>> Basically I wanted to point out discrepancy in name node status page and hadoop
>>> dfs -dus. In my case, earlier one reports DFS usage as 1TB and later
>>> one reports it to be 35GB. What are the factors that can cause this
>>> difference? And why is just 35GB data causing DFS to hit its limits?
>>>
>>>
>>>
>>>
>>> On 14-Apr-2014, at 8:31 am, Biswajit Nayak <biswajit.nayak@inmobi.com>
>>> wrote:
>>>
>>> Hi Saumitra,
>>>
>>> Could you please check the non-dfs usage. They also contribute to
>>> filling up the disk space.
>>>
>>>
>>>
>>> ~Biswa
>>> -----oThe important thing is not to stop questioning o-----
>>>
>>>
>>> On Mon, Apr 14, 2014 at 1:24 AM, Saumitra <saumitra.official@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> We are running HDFS on 9-node hadoop cluster, hadoop version is 1.2.1.
>>>> We are using default HDFS block size.
>>>>
>>>> We have noticed that disks of slaves are almost full. From name node’s
>>>> status page (namenode:50070), we could see that disks of live nodes are 90%
>>>> full and DFS Used% in cluster summary page  is ~1TB.
>>>>
>>>> However hadoop dfs -dus / shows that file system size is merely 38GB.
>>>> 38GB number looks to be correct because we keep only few Hive tables and
>>>> hadoop’s /tmp (distributed cache and job outputs) in HDFS. All other data
>>>> is cleaned up. I cross-checked this from hadoop dfs -ls. Also I think
>>>> that there is no internal fragmentation because the files in our Hive
>>>> tables are well-chopped in ~50MB chunks. Here are last few lines of
>>>> hadoop fsck / -files -blocks
>>>>
>>>> Status: HEALTHY
>>>>  Total size: 38086441332 B
>>>>  Total dirs: 232
>>>>  Total files: 802
>>>>  Total blocks (validated): 796 (avg. block size 47847288 B)
>>>>  Minimally replicated blocks: 796 (100.0 %)
>>>>  Over-replicated blocks: 0 (0.0 %)
>>>>  Under-replicated blocks: 6 (0.75376886 %)
>>>>  Mis-replicated blocks: 0 (0.0 %)
>>>>  Default replication factor: 2
>>>>  Average block replication: 3.0439699
>>>>  Corrupt blocks: 0
>>>>  Missing replicas: 6 (0.24762692 %)
>>>>  Number of data-nodes: 9
>>>>  Number of racks: 1
>>>> FSCK ended at Sun Apr 13 19:49:23 UTC 2014 in 135 milliseconds
>>>>
>>>>
>>>> My question is that why disks of slaves are getting full even though
>>>> there are only few files in DFS?
>>>>
>>>
>>>
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>>> the use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete transmission
>>> of the information contained in this communication nor for any delay in its
>>> receipt.
>>>
>>>
>>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>>
>
>
>
> --
> --Regards
>   Sandeep Nemuri
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message