hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5727) introduce a self-maintain io queue handling mechanism
Date Wed, 08 Jan 2014 04:07:53 GMT
Liang Xie created HDFS-5727:
-------------------------------

             Summary: introduce a self-maintain io queue handling mechanism
                 Key: HDFS-5727
                 URL: https://issues.apache.org/jira/browse/HDFS-5727
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: datanode
    Affects Versions: 3.0.0
            Reporter: Liang Xie
            Assignee: Liang Xie


Currently the datanode read/write SLA is dfficult to be ganranteed for HBase online requirement.
One of major reasons is we don't support io priority or io reqeust reorder inside datanode.
I proposal introducing a self-maintain io queue mechanism to handle io request priority. Image
there're lots of concurrent read/write reqeust from HBase side, and a background datanode
block scanner is running(default is every 21 days, IIRC) just in time, then the HBase read/write
99% or 99.9% percentile latency would be vulnerable despite we have a bg thread throttling...
the reorder stuf i have not thought clearly enough, but definitely the reorder in the queue
in the app side would beat the currently relying OS's io queue merge.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message