hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
Date Wed, 22 Jan 2014 09:39:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878459#comment-13878459

Liang Xie commented on HDFS-5776:

Added a bit codes into YCSB to get 99.9thPercentile latency, here is the result:
[READ], 95thPercentileLatency(us), 115973
[READ], 99thPercentileLatency(us), 124829
[READ], 99.9thPercentileLatency(us), 217892
dfs.dfsclient.hedged.read.threshold.millis=600000ms(this's equals to disable hedged read feature,
since my test duration is 600s):
[READ], 95thPercentileLatency(us), 149355
[READ], 99thPercentileLatency(us), 256987
[READ], 99.9thPercentileLatency(us), 418950
In practice, maybe we should set the threshold equals to 95/99th percentile latency, here
just a test to make the difference more obviously...

> Support 'hedged' reads in DFSClient
> -----------------------------------
>                 Key: HDFS-5776
>                 URL: https://issues.apache.org/jira/browse/HDFS-5776
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt,
> This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size"
to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics,
we could export the interested metric valus into client system(e.g. HBase's regionserver metric).
> The core logic is in pread code path, we decide to goto the original fetchBlockByteRange
or the new introduced fetchBlockByteRangeSpeculative per the above config items.

This message was sent by Atlassian JIRA

View raw message