hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhenxiao Luo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-4412) Support HDFS IO throttling
Date Wed, 16 Jan 2013 19:52:12 GMT
Zhenxiao Luo created HDFS-4412:
----------------------------------

             Summary: Support HDFS IO throttling
                 Key: HDFS-4412
                 URL: https://issues.apache.org/jira/browse/HDFS-4412
             Project: Hadoop HDFS
          Issue Type: New Feature
            Reporter: Zhenxiao Luo


When an applications upload/download files from/to HDFS clusters, it would be nice if the
IO could be throttled so that they won't go beyond the specified maximum bandwidth.

Two options to implement this IO throttling:

#1. IO Throttling happens at the FSDataInputStream and FSDataOutputStream level.

Add an IO Throttler to FSDataInputStream/FSDataOutputStram, and whenever an read/write happens,
throttle it first(if throttler is set), then do the actual read/write.

We may need to add new FileSystem apis to take an IO throttler as input parameter.

#2. IO Throttling happens at the application level.

Instead of changing the FSDataInputStream/FSDataOutputStream, all IO throttling is done at
the application level.

In this approach, FileSystem api remains unchanged.

Either case, an IO throttler interface is needed, which has a:
public void throttle(long numOfBytes);

The current DataTransferThrottler could be an implementation of this IO throttler interface.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message