hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5727) introduce a self-maintaining io queue handling mechanism
Date Wed, 08 Jan 2014 14:17:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13865467#comment-13865467
] 

Richard Chen commented on HDFS-5727:
------------------------------------

interesting but if you can improve your language further, you will help the audience to better
understand what you intend to do. My team is working on something similar to that. I am thinking
of adding your problem into our design scope. We can certainly collaborate on this. Let me
know your thoughts.

> introduce a self-maintaining io queue handling mechanism
> --------------------------------------------------------
>
>                 Key: HDFS-5727
>                 URL: https://issues.apache.org/jira/browse/HDFS-5727
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>
> Currently the datanode read/write SLA is difficult to be guaranteed for HBase online
requirement. One of major reasons is we don't support io priority or io request reorder inside
datanode.
> I propose introducing a self-maintain io queue mechanism to handle io request priority.
Imaging there're lots of concurrent read/write requests from HBase side, and a background
datanode block scanner is running(default is every 21 days, IIRC) just in time, then the HBase
read/write 99% or 99.9% percentile latency would be vulnerable despite we have a bg thread
throttling...
> the reorder stuff i have not thought clearly enough, but definitely the reorder in the
queue in the app side would beat the currently relying OS's io queue merge.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message