hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Denis Petrov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4412) Support HDFS IO throttling
Date Fri, 01 Mar 2013 20:07:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590895#comment-13590895
] 

Denis Petrov commented on HDFS-4412:
------------------------------------

I am trying to avoid a problem with "hot spots" in the disk IO.
In particular, during major compaction of Accumulo tablets (https://issues.apache.org/jira/browse/ACCUMULO-1128).

This problem is difficult to solve at the application level, because actual disk IO can be
(and often is) performed on another server, not on the server which runs the Accumulo tablet
server doing the compaction.
Two of three compactions running on different servers can result in heavy disk IO on the same
HDFS datanode resulting in degradation of query performance and latency.
                
> Support HDFS IO throttling
> --------------------------
>
>                 Key: HDFS-4412
>                 URL: https://issues.apache.org/jira/browse/HDFS-4412
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Zhenxiao Luo
>
> When an applications upload/download files from/to HDFS clusters, it would be nice if
the IO could be throttled so that they won't go beyond the specified maximum bandwidth.
> Two options to implement this IO throttling:
> #1. IO Throttling happens at the FSDataInputStream and FSDataOutputStream level.
> Add an IO Throttler to FSDataInputStream/FSDataOutputStram, and whenever an read/write
happens, throttle it first(if throttler is set), then do the actual read/write.
> We may need to add new FileSystem apis to take an IO throttler as input parameter.
> #2. IO Throttling happens at the application level.
> Instead of changing the FSDataInputStream/FSDataOutputStream, all IO throttling is done
at the application level.
> In this approach, FileSystem api remains unchanged.
> Either case, an IO throttler interface is needed, which has a:
> public void throttle(long numOfBytes);
> The current DataTransferThrottler could be an implementation of this IO throttler interface.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message