hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akmal Abbasov <akmal.abba...@icloud.com>
Subject Re: HDFS datanode used space is increasing without any writes
Date Thu, 30 Jul 2015 10:06:58 GMT
I am running HBase snapshot exporting, but I stopped it, and still the capacity used is increasing.

Here you can see that it is increased to 60 GB, and mostly it is because of 1 detanode.
Moreover I am checking the directories sizes using bin/hdfs dfs -du -h /
and while the HDFS used capacity is increasing, the directories sizes in HDFS is not changing.
Any ideas?

p.s. I started a hdfs balancer several days ago, then stopped it after running for about 10
minutes, after reading that it is not a good idea to run it on the same cluster with HBase.
Could it be because of this?

Thank you.


> On 28 Jul 2015, at 19:08, Harshit Mathur <mathursharp@gmail.com> wrote:
> 
> Are there any map reduce jobs running?
> 
> On Jul 28, 2015 10:11 PM, "Akmal Abbasov" <akmal.abbasov@icloud.com <mailto:akmal.abbasov@icloud.com>>
wrote:
> Hi, I’m observing strange behaviour in HDFS/HBase cluster.
> The disk space of one of datanodes is increasing very fast even when there are no write
requests. 
> It is 8GB per hour in average. Here is the graph which shows it.
> <screenshot.png>
> I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1.
> 
> And this is logs from the node
> 2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.32.1.12:50010 <http://10.32.1.12:50010/>, dest: /10.32.0.140:38699 <http://10.32.0.140:38699/>,
bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID:
6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663,
duration: 17759797
> 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: /10.0.0.21:60540 <http://10.0.0.21:60540/>
dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>
> 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.0.0.21:59054 <http://10.0.0.21:59054/>, dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>,
bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238,
duration: 3600203675041
> 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE
terminating
> 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:6099ms (threshold=300ms)
> 2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner:
BlockPool BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0,
missing block files:0, missing blocks in memory:0, mismatched blocks:0
> 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: /10.32.1.12:36998 <http://10.32.1.12:36998/>
dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>
> 2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.32.1.12:36150 <http://10.32.1.12:36150/>, dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>,
bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442,
duration: 3601152263901
> 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE
terminating
> 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: /10.0.0.19:35851 <http://10.0.0.19:35851/>
dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>
> 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.0.0.20:40176 <http://10.0.0.20:40176/>, dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>,
bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443,
duration: 3600482062810
> 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[]
terminating
> 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:7884ms (threshold=300ms)
> 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:4362ms (threshold=300ms)
> 2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync
took 3160ms (threshold=300ms), isSync:false, flushTotalNanos=3160364203ns
> 2015-07-28 16:13:29,968 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:659ms (threshold=300ms)
> 2015-07-28 16:18:33,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-439084760-10.32.0.180-1387281790961:blk_1075311717_1574453 src: /10.0.0.20:41527 <http://10.0.0.20:41527/>
dest: /10.32.1.12:50010 <http://10.32.1.12:50010/>
> 2015-07-28 16:18:38,926 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:1767ms (threshold=300ms)
> 2015-07-28 16:28:40,580 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:4099ms (threshold=300ms)
> 
> What could be the cause of this?
> Thank you.
> 
> 


Mime
View raw message