hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6450) Support non-positional hedged reads in HDFS
Date Wed, 16 Jul 2014 18:46:10 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063892#comment-14063892

Colin Patrick McCabe commented on HDFS-6450:

Yeah... if the first read, done with the old block reader, doesn't finish first, then I think
we should forget about that block reader... even if it's a {{BlockReaderLocal}} and the new
one is remote.  After all, slow or misbehaving local disks are one of the main problems we're
trying to cover up with hedged reads.  As you pointed out, we need to close the old block
reader in this situation.

I think from an implementation point of view, you might consider unsetting {{DFSInputStream#blockReader}}
at the beginning of the hedged read (set to null and move the old block reader into the first
future).  Then have the "winning" future set it to the block reader that it used.

There will be some performance regression just due to using threads and futures here, instead
of just doing the read from the same thread that needs the data.  Passing data between threads
is slower because you might be going between CPU cores.  I don't know if there's really a
good way to address this without doing something like HDFS-6695, which is out of scope for
this change.  I think in the short term, applications will have to turn on non-positional
hedged reads explicitly, and accept some small loss in throughput for a major reduction in
long-tail latency.

> Support non-positional hedged reads in HDFS
> -------------------------------------------
>                 Key: HDFS-6450
>                 URL: https://issues.apache.org/jira/browse/HDFS-6450
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.4.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Liang Xie
>         Attachments: HDFS-6450-like-pread.txt
> HDFS-5776 added support for hedged positional reads.  We should also support hedged non-position
reads (aka regular reads).

This message was sent by Atlassian JIRA

View raw message