hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4412) Support HDFS IO throttling
Date Thu, 17 Jan 2013 19:10:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556481#comment-13556481
] 

Colin Patrick McCabe commented on HDFS-4412:
--------------------------------------------

I think it makes sense to create some kind of throttler class that wraps a generic {{OutputStream}}.
 That doesn't need to be in HDFS-- in fact, that code should probably go in common.
                
> Support HDFS IO throttling
> --------------------------
>
>                 Key: HDFS-4412
>                 URL: https://issues.apache.org/jira/browse/HDFS-4412
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Zhenxiao Luo
>
> When an applications upload/download files from/to HDFS clusters, it would be nice if
the IO could be throttled so that they won't go beyond the specified maximum bandwidth.
> Two options to implement this IO throttling:
> #1. IO Throttling happens at the FSDataInputStream and FSDataOutputStream level.
> Add an IO Throttler to FSDataInputStream/FSDataOutputStram, and whenever an read/write
happens, throttle it first(if throttler is set), then do the actual read/write.
> We may need to add new FileSystem apis to take an IO throttler as input parameter.
> #2. IO Throttling happens at the application level.
> Instead of changing the FSDataInputStream/FSDataOutputStream, all IO throttling is done
at the application level.
> In this approach, FileSystem api remains unchanged.
> Either case, an IO throttler interface is needed, which has a:
> public void throttle(long numOfBytes);
> The current DataTransferThrottler could be an implementation of this IO throttler interface.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message