hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6450) Support non-positional hedged reads in HDFS
Date Wed, 08 Mar 2017 18:06:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901684#comment-15901684

Inigo Goiri commented on HDFS-6450:

In our own scenario, some Datanodes are very likely to be much slower than others.
For this reason, we need non-positional hedged reads.

[~thinktaocs] has worked on porting and extending the original patch in this JIRA to work
in 2.7 and support multiple hedged reads.
In our prototype, we use Futures and we allow cancelling/killing slow reads and so on.
We've been using it for the last month or so and the improvements are very significant.
We've been able to reduce our the tail time of our reads by more than 2x.

Would it be possible for [~thinktaocs] to be assigned to this JIRA and upload our prototype?
Based on this patch we can start the discussion on the approach.

> Support non-positional hedged reads in HDFS
> -------------------------------------------
>                 Key: HDFS-6450
>                 URL: https://issues.apache.org/jira/browse/HDFS-6450
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.4.0
>            Reporter: Colin P. McCabe
>            Assignee: Liang Xie
>         Attachments: HDFS-6450-like-pread.txt
> HDFS-5776 added support for hedged positional reads.  We should also support hedged non-position
reads (aka regular reads).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message