hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
Date Tue, 28 Jan 2014 09:07:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13883928#comment-13883928

Jing Zhao commented on HDFS-5776:

bq.  it's more flexible if we provide instance level disable/enable APIs, so we can archive
to use the hbase shell script to control the switch per dfs client instance, that'll be cooler
I still have some concern about the current implementation: 
1) we do not check threadpool in enableHedgedReads. This makes it possible that isHedgedReadsEnabled()
returns true while hedged read is actually not enabled.
2) DFSClient#setThreadsNumForHedgedReads allows users to keep changing the size of the thread
To provide instance level disable/enable APIs, I think maybe we can do the following:
1) Read the thread pool size configuration only when initializing the thread pool, and the
size should be >0 and cannot be changed.
2) Add an "Allow-Hedged-Reads" configuration. Each DFSClient instance reads this configuration,
and if it is true, checks and initializes the thread pool if necessary. Users can turn on/off
the switch using the enable/disable methods. In the enable method, we check and initialize
the thread pool if necessary.

What do you think [~xieliang007]?

> Support 'hedged' reads in DFSClient
> -----------------------------------
>                 Key: HDFS-5776
>                 URL: https://issues.apache.org/jira/browse/HDFS-5776
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-5776-v10.txt, HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt,
HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776-v9.txt,
> This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size"
to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics,
we could export the interested metric valus into client system(e.g. HBase's regionserver metric).
> The core logic is in pread code path, we decide to goto the original fetchBlockByteRange
or the new introduced fetchBlockByteRangeSpeculative per the above config items.

This message was sent by Atlassian JIRA

View raw message