hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lohit <lohit.vijayar...@gmail.com>
Subject HDFS read/write data throttling
Date Mon, 11 Nov 2013 18:59:52 GMT
Hello Devs,

Wanted to reach out and see if anyone has thought about ability to throttle
data transfer within HDFS. One option we have been thinking is to throttle
on a per FileSystem basis, similar to Statistics in FileSystem. This would
mean anyone with handle to HDFS/Hftp will be throttled globally within JVM.
Right value to come up for this would be based on type of hardware we use
and how many tasks/clients we allow.

On the other hand doing something like this at FileSystem layer would mean
many other tasks such as Job jar copy, DistributedCache copy and any hidden
data movement would also be throttled. We wanted to know if anyone has had
such requirement on their clusters in the past and what was the thinking
around it. Appreciate your inputs/comments

Have a Nice Day!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message