hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6450) Support non-positional hedged reads in HDFS
Date Tue, 15 Jul 2014 06:32:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061760#comment-14061760

Liang Xie commented on HDFS-6450:

After a deep looking, it's kind of hard to reuse/maintain block reader as before. 
In pread(), we don't have this trouble, because we always create new block reader.
In read(), if we want to support hedged read ability, in general:
1) first read(r1) using the old block reader if possible, then wait hedged read timeout setting
2) second read(r2) must create a new block reader, and submit into thread pool
3) wait the first completed task, and return final read result to client side.  Here we need
to set(remember) this task's block reader to DFIS's block reader variable, and should keep
it open,  but we also need to close the other block reader to avoid leak.

Another thing need to know is that if we remember the faster block reader, if it's a remote
block reader, then the following read() will bypass local read in the following r1 operations...
Any thought ? [~cmccabe], [~saint.ack@gmail.com] ... 

> Support non-positional hedged reads in HDFS
> -------------------------------------------
>                 Key: HDFS-6450
>                 URL: https://issues.apache.org/jira/browse/HDFS-6450
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.4.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Liang Xie
>         Attachments: HDFS-6450-like-pread.txt
> HDFS-5776 added support for hedged positional reads.  We should also support hedged non-position
reads (aka regular reads).

This message was sent by Atlassian JIRA

View raw message