hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From raymond <rgbbo...@163.com>
Subject Best way to throttle hadoop network bandwidth?
Date Mon, 11 Apr 2016 11:20:04 GMT

We have a need to throttle hadoop network bandwidth across nodes. You might ask why, the reason
is that we need to limit the overall bandwidth consumed by Hadoop cluster so that it can live
happily with other servers located under the same switches . we know it can also be solved
by rearrange server and carefully plan the network topology, or a reasonable config on the
switch might works. But we have trouble to apply those approaching. For now, we need a solution
to cut down the network bandwidth consumed by hadoop to say 60MB/s instead of the full bandwidth
120MB/s .

We have tried many approaches, but none of them works well.

1. we try to use TC to throttle network interface’s bandwidth.

we tried something like :

tc qdisc add dev eth0 root tbf rate 480Mbit burst 1M latency 10ms , on centos6.x, we hope
by control every node’s outbound bandwidth to reach our goal.

but it seems that it doesn’t works quite well with hadoop workload. A simple scp test run’s
well, but with a heavy loaded hadoop cluster, we encounter several random kernel crashes in
just a couple hours, and the core dump show something like:

<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000010


<0>Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2 

in the latter case, it seems that a dequeue operation on qdisc last too long time, also the
interrupt are masked. then trigger the watchdog.

it seems like a kernel’s bug ( not quite sure, but some clue shows that 3.1 kernel fixed
it)   or we don’t made the right config. or hadoop workload just doesn’t work well under
this case. we know a lot of people using tc to do the traffic control, but we just failed
to make it run long enough.

2. we try to use iptables to limit the bandwidth like:

iptables -A OUTPUT -m limit  --limit 10000/sec --limit-burst 10000 -j ACCEPT
iptables -A OUTPUT -j REJECT

however, the —limit parameter only accept numbers smaller than 10k , thus it can only provide
15MB/s bandwidth. 

3. we try to tune tcp data transfer win size by adjust tcp sending / receiving buffer etc.

It can somehow throttle the bandwidth, but it is not accurate, there is no obvious relation
between the buffer size and transfer speed. And I believe it depends on cpu load and the actual
throughput can vary a lot. So I doubt this way can work well.

4. turn to Hadoop itself, 

there were several issue related to data node bandwidth control 

https://issues.apache.org/jira/browse/HDFS-9796 <https://issues.apache.org/jira/browse/HDFS-9796>
https://issues.apache.org/jira/browse/HDFS-7270 <https://issues.apache.org/jira/browse/HDFS-7270>
https://issues.apache.org/jira/browse/HDFS-7265 <https://issues.apache.org/jira/browse/HDFS-7265>

but they seems like applied on disk IO etc. none of them can solve our problem.


https://issues.apache.org/jira/browse/HDFS-8683 <https://issues.apache.org/jira/browse/HDFS-8683>

this one aim to do flow control on top of HTTP2, and not yet done.

Anyway, seems no quick solution for our problem. But I believe this should be able to be done
from the data node side. 

So, Can anyone give us some advices? Any?


View raw message