hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5727) introduce a self-maintaining io queue handling mechanism
Date Wed, 08 Jan 2014 13:51:57 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Richard Chen updated HDFS-5727:
-------------------------------

    Summary: introduce a self-maintaining io queue handling mechanism  (was: introduce a self-maintain
io queue handling mechanism)

> introduce a self-maintaining io queue handling mechanism
> --------------------------------------------------------
>
>                 Key: HDFS-5727
>                 URL: https://issues.apache.org/jira/browse/HDFS-5727
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>
> Currently the datanode read/write SLA is difficult to be guaranteed for HBase online
requirement. One of major reasons is we don't support io priority or io request reorder inside
datanode.
> I propose introducing a self-maintain io queue mechanism to handle io request priority.
Imaging there're lots of concurrent read/write requests from HBase side, and a background
datanode block scanner is running(default is every 21 days, IIRC) just in time, then the HBase
read/write 99% or 99.9% percentile latency would be vulnerable despite we have a bg thread
throttling...
> the reorder stuff i have not thought clearly enough, but definitely the reorder in the
queue in the app side would beat the currently relying OS's io queue merge.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message