hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nagalingam, Karthikeyan" <KARTHIKEYAN.Nagalin...@netapp.com>
Subject slow blockreceiver write data to disk
Date Fri, 17 Jul 2015 14:05:41 GMT
Hi,

The testdfsio write operation takes longer time in a 8 datanode with 48GB RAM and 16 CPU Core
with 10GigE. The 10GigE network port utilization 1.5 to 2 gigabits/sec for each datanode during
write operation.

xfs filesystem config:

parted -s /dev/sdb mklabel gpt mkpart /dev/sdb1 xfs 6144s 10.0TB
/sbin/mkfs.xfs -f -L DISK1 -l size=128m,sunit=256,lazy-count=1 -d su=512k,sw=6 -r extsize=256k
/dev/sdb1
mkdir /disk1
mount /disk1
/sbin/blockdev --setra 1024 /dev/sdb
df -h /disk1"

/etc/fstab config:

LABEL=DISK1   /disk1    xfs     allocsize=128m,noatime,nobarrier,nodiratime     0       0

The /var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-hadoop1.com.log.out shows

For dfs.blocksize=128m

2015-07-14 17:47:16,391 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:614ms (threshold=300ms)
2015-07-14 17:47:16,400 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:623ms (threshold=300ms)
2015-07-14 17:47:16,401 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:548ms (threshold=300ms)
2015-07-14 17:47:16,420 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write data to disk cost:567ms (threshold=300ms)

For dfs.blocksize=512m

2015-07-17 09:46:28,999 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write packet to mirror took 408ms (threshold=300ms)
2015-07-17 09:46:28,999 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write packet to mirror took 448ms (threshold=300ms)
2015-07-17 09:46:29,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write packet to mirror took 409ms (threshold=300ms)
2015-07-17 09:46:29,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver
write packet to mirror took 451ms (threshold=300ms)

Please throw some light to fix the above issue

Regards,
Karthikeyan Nagalingam
TME (Big Data Analytics)
Mobile: 919-376-6422


Mime
View raw message