hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
Date Tue, 02 Jun 2015 09:09:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568816#comment-14568816
] 

Walter Su commented on HDFS-8328:
---------------------------------


1a. The first while loop should guarded with {{!used.get(m)}}. m starting from {{stripedReaders.size()}}
doesn't mean it's not used.
{code}
    private int scheduleNewRead(BitSet used) {
      StripedReader reader = null;
      int m = stripedReaders.size();
      while (reader == null && m < sources.length) {
        if(!used.get(m)) {
          reader = addStripedReader(m, positionInBlock);
          if (getReadLength(liveIndices[m]) > 0) {
            if (reader.blockReader == null) {
              reader = null;
            }   
          }   
        }   
        m++;
      }   
{code}
1b. if getReadLength(liveIndices\[m\]) == 0, we only have to keep one {{return}} statement
at second for loop.

2.paddingBufferToLen(..) for multiple zeroStripeBuffers[] is wasting time. One zeroStripeBuffer
is enough.
{code}
private void recoverTargets(int[] success, boolean[] targetsStatus,
...
if (success.length < dataBlkNum) {
for (int i = 0; i < zeroStripeBuffers.length; i++) {
  ByteBuffer buffer = zeroStripeBuffers[i];
  paddingBufferToLen(buffer, toRecoverLen);
  int index = covertIndex4Decode(zeroStripeIndices[i], dataBlkNum,
      parityBlkNum);
  inputs[index] = (ByteBuffer)buffer.flip();
}
{code}
We prepare one zeroStripeBuffer. Then use ByteBuffer.slice() to create a "mirror" buffer.
It saves memory and time.

> Follow-on to update decode for DataNode striped blocks reconstruction
> ---------------------------------------------------------------------
>
>                 Key: HDFS-8328
>                 URL: https://issues.apache.org/jira/browse/HDFS-8328
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8328-HDFS-7285.001.patch, HDFS-8328-HDFS-7285.002.patch
>
>
> Current the decode for DataNode striped blocks reconstruction is a workaround, we need
to update it after the decode fix in HADOOP-11847.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message