hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
Date Mon, 01 Jun 2015 22:00:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568107#comment-14568107
] 

Zhe Zhang commented on HDFS-8328:
---------------------------------

Thanks for the update Yi! The patch looks good overall. I have the following questions:

# The use of {{targetsStatus}} is a little tricky. Let me know if I'm understanding it correctly:
#* If initial connection to a target DN (chosen by NN) is unsuccessful, {{getErasedIndices}}
will not add the index in {{erasedIndices}}; therefore the block will not be reconstructed.
Should we send an ack to NN so the block can come back to {{UnderReplicatedBlocks}}?
#* In each iteration of {{while (positionInBlock < firstStripedBlockLength)}}, {{targetsStatus}}
is reset to reflect if the transfer to each target is successful. So when an element is flipped
from {{true}} to {{false}}, it will never be flipped back.
#* If the above is correct, maybe let's add some Javadoc and TODO if necessary (for the NN
ack part)?
# {{zeroStripeBuffers}} is created based on whether the entire internal block is empty. Ideally
we should have such an array in each iteration of {{while (positionInBlock < firstStripedBlockLength)}}
so we can avoid issuing read requests as much as possible. I guess this can be a TODO for
follow-on.
# The updated {{scheduleNewRead}} is not so easy to follow. 
#* In the first loop, is it assumed that the reader indices in {{stripedReaders}} begin from
0 and are consecutive?
{code}
      int m = stripedReaders.size();
      while (reader == null && m < sources.length) {
        reader = addStripedReader(m, positionInBlock);
        xxx
{code}
#* I feel the method does the right thing but haven't fully understood it yet. It would be
very helpful to have comments for steps 1, 2, 3 in the method.
# The argument and return value of {{readMinimumStripedData4Recovery}} are not trivial to
understand. Maybe add them to the Javadoc?
# Since we are fetching mostly fixed size chunks, it's ideal to reuse buffers instead of allocating
new ones in {{addStripedReader}}. [~walter.k.su] has a good [analysis | https://issues.apache.org/jira/browse/HDFS-8481?focusedCommentId=14564053&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14564053]
under HDFS-8481 on the GC issue.

> Follow-on to update decode for DataNode striped blocks reconstruction
> ---------------------------------------------------------------------
>
>                 Key: HDFS-8328
>                 URL: https://issues.apache.org/jira/browse/HDFS-8328
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8328-HDFS-7285.001.patch, HDFS-8328-HDFS-7285.002.patch
>
>
> Current the decode for DataNode striped blocks reconstruction is a workaround, we need
to update it after the decode fix in HADOOP-11847.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message