Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 24 Jan 2014 07:16:41 +0000 (UTC)
From: "Liang Xie (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12688930.1389757190823.5692.1390547801956@arcas>
In-Reply-To: <JIRA.12688930.1389757190823@arcas>
References: <JIRA.12688930.1389757190823@arcas>
Subject: [jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880783#comment-13880783 ] 

Liang Xie commented on HDFS-5776:
---------------------------------

v8 add the enoughNodesForHedgedRead() function to sanity check, [~saint.ack@gmail.com]'s comments is great, we definitely need a switch per DFSClient instance. 
bq. the number of threads config needs to be NumberOfHBaseOpenFiles X 2 else the feature will not work for all files
still works, but probably lots of requests will execute in current thread, that means no latency benefit from hedged read feature.  this's is a good requirement that we need a per client instance's switch, such that we can let some instances use this feature,  we can control it on demand, right ? :)

> Support 'hedged' reads in DFSClient
> -----------------------------------
>
>                 Key: HDFS-5776
>                 URL: https://issues.apache.org/jira/browse/HDFS-5776
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric).
> The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items.


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)