hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akmal Abbasov <akmal.abba...@icloud.com>
Subject HDFS datanode used space is increasing without any writes
Date Tue, 28 Jul 2015 16:41:36 GMT
Hi, I’m observing strange behaviour in HDFS/HBase cluster.
The disk space of one of datanodes is increasing very fast even when there are no write requests.

It is 8GB per hour in average. Here is the graph which shows it.

I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1.

And this is logs from the node
2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663,
duration: 17759797
2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450
src: /10.0.0.21:60540 dest: /10.32.1.12:50010
2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238,
duration: 3600203675041
2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE
terminating
2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:6099ms (threshold=300ms)
2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0, missing
block files:0, missing blocks in memory:0, mismatched blocks:0
2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451
src: /10.32.1.12:36998 dest: /10.32.1.12:50010
2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442,
duration: 3601152263901
2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE
terminating
2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452
src: /10.0.0.19:35851 dest: /10.32.1.12:50010
2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35,
offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443,
duration: 3600482062810
2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[]
terminating
2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:7884ms (threshold=300ms)
2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:4362ms (threshold=300ms)
2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync
took 3160ms (threshold=300ms), isSync:false, flushTotalNanos=3160364203ns
2015-07-28 16:13:29,968 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:659ms (threshold=300ms)
2015-07-28 16:18:33,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311717_1574453
src: /10.0.0.20:41527 dest: /10.32.1.12:50010
2015-07-28 16:18:38,926 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:1767ms (threshold=300ms)
2015-07-28 16:28:40,580 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:4099ms (threshold=300ms)

What could be the cause of this?
Thank you.



Mime
View raw message