hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
Date Wed, 22 Apr 2015 23:56:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508177#comment-14508177
] 

Jing Zhao commented on HDFS-8033:
---------------------------------

Thanks for updating the patch, Zhe! Some comments on the 002 patch:
# I think we do not need to override the blockSeekTo method. The return type of blockSeekTo
is a DataNodeInfo, which does not suit striped stream very well. Thus I guess we only need
to define a new {{seekToNextBlock}} method which have the same functionality but different
signature.
# To me {{currentNode}} and {{blockReader}} should be internal states only used by {{DFSInputStream}}.
{{DFSStripedInputStream}} should only use its own reader/DN list, instead of setting {{currentNode}}
and {{blockReader}} and calling {{DFSInputStream#readBuffer}}. We can define a different version
of {{readBuffer}} method which utilizes the reader list to read. And this new {{readBuffer}}
method does not need to override the one in DFSInputStream.
# Another reason that we do not reuse the {{DFSInutStream#readBuffer}} is that the retry logic
is different. From striped inputstream we do not need to try another source when hitting any
failure.
# We need more tests. Now instead of injecting ec blocks, we can directly use DFSStripedOutputstream
for writing. And then we can compare the reading results (of both stateful read and pread)
with the original written content.

> Erasure coding: stateful (non-positional) read from files in striped layout
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-8033
>                 URL: https://issues.apache.org/jira/browse/HDFS-8033
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message