Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 523A81027D for ; Wed, 8 Jan 2014 14:30:19 +0000 (UTC) Received: (qmail 95608 invoked by uid 500); 8 Jan 2014 14:18:08 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 95548 invoked by uid 500); 8 Jan 2014 14:18:03 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 95450 invoked by uid 99); 8 Jan 2014 14:17:58 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jan 2014 14:17:58 +0000 Date: Wed, 8 Jan 2014 14:17:58 +0000 (UTC) From: "Richard Chen (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-5727) introduce a self-maintaining io queue handling mechanism MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13865467#comment-13865467 ] Richard Chen commented on HDFS-5727: ------------------------------------ interesting but if you can improve your language further, you will help the audience to better understand what you intend to do. My team is working on something similar to that. I am thinking of adding your problem into our design scope. We can certainly collaborate on this. Let me know your thoughts. > introduce a self-maintaining io queue handling mechanism > -------------------------------------------------------- > > Key: HDFS-5727 > URL: https://issues.apache.org/jira/browse/HDFS-5727 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode > Affects Versions: 3.0.0 > Reporter: Liang Xie > Assignee: Liang Xie > > Currently the datanode read/write SLA is difficult to be guaranteed for HBase online requirement. One of major reasons is we don't support io priority or io request reorder inside datanode. > I propose introducing a self-maintain io queue mechanism to handle io request priority. Imaging there're lots of concurrent read/write requests from HBase side, and a background datanode block scanner is running(default is every 21 days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile latency would be vulnerable despite we have a bg thread throttling... > the reorder stuff i have not thought clearly enough, but definitely the reorder in the queue in the app side would beat the currently relying OS's io queue merge. -- This message was sent by Atlassian JIRA (v6.1.5#6160)