hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10410) Support ioprio_set in NativeIO
Date Thu, 20 Mar 2014 19:40:48 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942185#comment-13942185

Andrew Wang commented on HADOOP-10410:

Hey Haohui, I can speak a little on this,

One nifty thing you can do with ioprio is set priorities on a per-thread basis. This lets
you do differentiated service among the DataXceivers in a datanode based on (for example)
a QoS field or a username.

Regarding capabilities, I think you only need CAP_SYS_ADMIN to set things in the RT class.
You can still use the normal priority class as long as you're the owner of the process. The
ioprio_set page talks about this (http://man7.org/linux/man-pages/man2/ioprio_set.2.html)

I don't have my numbers from my ioprio prototype anymore, but I found pretty nice gains in
99th percentile latency after I zeroed out the disk queue. I didn't bother submitting it upstream
at the time since I thought the throughput hit wasn't worth it, but Liang said that latency
is his foremost concern.

> Support ioprio_set in NativeIO
> ------------------------------
>                 Key: HADOOP-10410
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10410
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: native
>    Affects Versions: 3.0.0, 2.4.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HADOOP-10410.txt
> It would be better to HBase application if HDFS layer provide a fine-grained IO request
priority. Most of modern kernel should support ioprio_set system call now.

This message was sent by Atlassian JIRA

View raw message