hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
Date Wed, 22 Apr 2015 00:43:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506131#comment-14506131
] 

Yi Liu edited comment on HDFS-8033 at 4/22/15 12:42 AM:
--------------------------------------------------------

Thanks Zhe for the response, and also thanks Jing for the review and good comments!   
Yes, it's a good idea to maintain a list of current DataNodes.

{quote}
DFSStripedInputStream#readBuffer does switch the blockReader. So after reading cell_0, we'll
switch to the next blockReader and read cell_1.
{quote}
{{DFSStripedInputStream#readBuffer}} can switch the blockReader, that's true, but it depends
on the caller to call, currently for one stateful read, {{DFSInputStream#readWithStrategy}}
only call it one time, so the result only contains part of the expected data.
I didn't see you do loop (parallelly) all blockReaders for the striped block in {{DFSStripedInputStream#readBuffer}}.

I have another comments, 
I think we need to change {{closeCurrentBlockReader}} to {{closeCurrentBlockReaders}} in {{DFSInputStream}}
and override it in {{DFSStripedInputStream}}, otherwise there is leak for {{blockReaders}}.


was (Author: hitliuyi):
Thanks Zhe for the response, and also thanks Jing for the review and good comments!   
Yes, it's a good idea to maintain a list of current DataNodes.

{quote}
DFSStripedInputStream#readBuffer does switch the blockReader. So after reading cell_0, we'll
switch to the next blockReader and read cell_1.
{quote}
{{DFSStripedInputStream#readBuffer}} can switch the blockReader, that's true, but it depends
on the caller to call, currently for one stateful read, {{DFSInputStream#readWithStrategy}}
only call it one time, so the result only contains part of the expected data.
I didn't see you do loop (parallelly) all blockReaders for the striped block in {{DFSStripedInputStream#readBuffer}}.

I have another comments, 
I think we need to change {{closeCurrentBlockReader}} to {{closeCurrentBlockReaders}} in {{DFSInputStream}}
and overwrite it in {{DFSStripedInputStream}}, otherwise there is leak for {{blockReaders}}.

> Erasure coding: stateful (non-positional) read from files in striped layout
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-8033
>                 URL: https://issues.apache.org/jira/browse/HDFS-8033
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message