hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5727) introduce a self-maintain io queue handling mechanism
Date Wed, 08 Jan 2014 04:07:53 GMT
Liang Xie created HDFS-5727:

             Summary: introduce a self-maintain io queue handling mechanism
                 Key: HDFS-5727
                 URL: https://issues.apache.org/jira/browse/HDFS-5727
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: datanode
    Affects Versions: 3.0.0
            Reporter: Liang Xie
            Assignee: Liang Xie

Currently the datanode read/write SLA is dfficult to be ganranteed for HBase online requirement.
One of major reasons is we don't support io priority or io reqeust reorder inside datanode.
I proposal introducing a self-maintain io queue mechanism to handle io request priority. Image
there're lots of concurrent read/write reqeust from HBase side, and a background datanode
block scanner is running(default is every 21 days, IIRC) just in time, then the HBase read/write
99% or 99.9% percentile latency would be vulnerable despite we have a bg thread throttling...
the reorder stuf i have not thought clearly enough, but definitely the reorder in the queue
in the app side would beat the currently relying OS's io queue merge.

This message was sent by Atlassian JIRA

View raw message