hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8319) Erasure Coding: support decoding for stateful read
Date Tue, 02 Jun 2015 21:25:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569792#comment-14569792

Zhe Zhang commented on HDFS-8319:

Great work Jing! {{StripeReader}} is a very good abstraction for p/np reads. The patch looks
pretty good to me. A few thoughts are listed below; some are suggested future work.
# Nit: we can move {{readToBuffer}} to be right after {{readCell}}, which is its only caller.
# We can rename {{prepareParityChunk}} to {{prepareAdditionalChunks}} because in pread we
will possibly issue additional data chunks to be fetched. Then we can move out the chunk-prep
code from {{initDecodeInputs}} either in this patch or as a follow-on (there are a few other
places where we can further consolidate the code anyway).
# In both read modes, our recovery algorithm is to issue all possible requests upon _first
failure_. We can think about if we want to support other less expensive algorithms. If not,
we can change the structure of the while loop in {{readStripe}} so that we don't call {{prepareDecodeInputs}}
and {{prepareParityChunk}} for each missing block.
# {{DFSStripedInputStream#copy}} could perhaps use a better name. The Javadoc is pretty clear
# We should either rename {{testWritePreadWithDNFailure}} or have the stateful read as a separate
test. Rename to {{testWriteReadWithDNFailure}} is easier. BTW the test failure is caused by
the order of parity and data blocks in the decode input buffers (see HDFS-8517). 
# As a follow-on, we can probably always convert byte array to byte buffer so we can further
consolidate {{divideOneStripe}} with {{divideByteRangeIntoStripes}} and the internal code
of {{StripingChunk}}; maybe we can also consolidate the 2 {{decode()}}.
# As a follow-on, we can consider making {{StripeReader}} static so DataNode can potentially
use it too.

> Erasure Coding: support decoding for stateful read
> --------------------------------------------------
>                 Key: HDFS-8319
>                 URL: https://issues.apache.org/jira/browse/HDFS-8319
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-8319.001.patch, HDFS-8319.002.patch
> HDFS-7678 adds the decoding functionality for pread. This jira plans to add decoding
to stateful read.

This message was sent by Atlassian JIRA

View raw message