hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
Date Tue, 21 Apr 2015 23:24:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506006#comment-14506006

Zhe Zhang commented on HDFS-8033:

Continuing with Jing's comments:
bq. In DFSInputStream, the reading state can be captured by blockReader, currentNode, etc.
When reading striped blocks, it may be more clear to define the state as the group of readers
and DataNodes.
Good point about maintaining a list of current DataNodes. This will be helpful to track bad
nodes in the parallel reading case. The patch already maintains {{blockReaders}}. The entire
list of {{blockReader}}s are forwarded to the next striped block group every time {{blockSeekTo}}
is called. I guess in the parallel stateful reading case we can just read from those {{blockReader}}s
in parallel.

bq. Thus instead of declare blockReader, currentNode, etc as protected and keep updating them
in DFSStripedInputStream, I guess it may be better to define DFSStripedInputStream's own states
(using a list of readers etc.). We may want to override the whole readWithStrategy method
in DFSStripedInputStream as well.
Yes, per the above discussion, we should also override {{readWithStrategy}} to handle {{currentNodes}}

> Erasure coding: stateful (non-positional) read from files in striped layout
> ---------------------------------------------------------------------------
>                 Key: HDFS-8033
>                 URL: https://issues.apache.org/jira/browse/HDFS-8033
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch

This message was sent by Atlassian JIRA

View raw message