hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georgi Ivanov <iva...@vesseltracker.com>
Subject Re: HDFS multiple dfs_data_dir disbalance
Date Wed, 22 Oct 2014 13:47:00 GMT
Thanks for the reply.

Unfortunately there is no extra data in this dir.

This if form the DN log :
2014-10-22 15:29:00,205 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: 
Added volume - /data/1/dfs/dn/current
2014-10-22 15:29:00,205 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: 
Added volume - /data/2/dfs/dn/current

Scheduling blk_8969115446695150692_1784945 file 
/data/2/dfs/dn/current/BP-1312742174-78.46.149.194-1359718879114/current/finalized/subdir44/subdir31/blk_8969115446695150692

for deletion

So i can see /data/2 is used.

These dirs are actually 2 different disks. I now remember that one of 
those died recently and was replaced.

I don't see errors with fsck .

Name: xx.xx.xx.xx:50010 (dn2.domain.com)
Hostname: dn2
Rack: /default
Decommission Status : Normal
Configured Capacity: 5651963387904 (5.14 TB)
DFS Used: 3548748439552 (3.23 TB)
Non DFS Used: 267148099584 (248.80 GB)
DFS Remaining: 1836066848768 (1.67 TB)
DFS Used%: 62.79%
DFS Remaining%: 32.49%
Last contact: Wed Oct 22 15:42:46 CEST 2014

As you can see here is another proof that /data/2 is used, as we still 
have 1.67TB free. If it was not used we would have ~0% free

So the dir is used, but it is not balanced.
I think this is because of the disk crash.
But isn't it supposed Hadoop to fix this ?
The disk was replaced few weeks ago...

Georgi

On 22.10.2014 15:05, Brahma Reddy Battula wrote:
> does /data1 is having non-hadoop data..? Please check for same..
> check admin report(hdfs dfsadmin -report) and fsck report (hdfs fsck /)
>
> Thinking that following might not
> a) /data/2 is not having the permission to write data where volumes tolerated configured
as 1
> b) /data/2 is not added after some time
>
>
>
> Thanks & Regards
> Brahma Reddy Battula
> ________________________________________
> From: Georgi Ivanov [ivanov@vesseltracker.com]
> Sent: Wednesday, October 22, 2014 5:17 PM
> To: user@hadoop.apache.org
> Subject: HDFS multiple dfs_data_dir disbalance
>
> Hi,
> My cluster is configured with 2 data dirs.
> /data/1
> /data/2
>
> Usually hadoop is balancing the utilization of these dirs.
> Now i have one node where /data/1 is 100% full and /data/2 is not.
>
> Is there anything i can do about this, as this results in failed
> mapppers/rdecers ?
>
> Georgi
>
>
>
>


Mime
View raw message