hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
Date Thu, 30 Jan 2014 00:24:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886072#comment-13886072

Jing Zhao commented on HDFS-5776:

Thanks for updating the patch, [~xieliang007] and [~stack].

[~stack], so the latest patch changes setThreadsNumForHedgedReads to private and aims to make
users unable to "change the thread number from client side dynamically". However, users can
still create their own configuration object, change the configuration for thread pool size,
create an DFSClient instance, and change the thread number? So I think we may want to make
it more clean here. Specifically,
# the first DFSClient who tries to enable the hedged read will initialize the thread pool
(in the DFSClient constructor or in the enable method), so that the enable can be a real enable
# changing of the thread pool size (if it is necessary) should still go through a setThreadsNumForHedgedReads
method (instead of the constructor of DFSClient), so that a client cannot silently change
the size of the thread pool

Besides, the current patch has not addressed the comment for enoughNodesForHedgedRead/chooseDataNode.

> Support 'hedged' reads in DFSClient
> -----------------------------------
>                 Key: HDFS-5776
>                 URL: https://issues.apache.org/jira/browse/HDFS-5776
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-5776-v10.txt, HDFS-5776-v11.txt, HDFS-5776-v12.txt, HDFS-5776-v12.txt,
HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt,
HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776-v9.txt, HDFS-5776.txt
> This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size"
to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics,
we could export the interested metric valus into client system(e.g. HBase's regionserver metric).
> The core logic is in pread code path, we decide to goto the original fetchBlockByteRange
or the new introduced fetchBlockByteRangeSpeculative per the above config items.

This message was sent by Atlassian JIRA

View raw message