hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
Date Wed, 15 Apr 2015 06:57:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495769#comment-14495769

Jing Zhao commented on HDFS-8120:

Thanks for the explanation, Bo.

bq. When DFSStripedOutputStream requests a block group from NN, it also doesn't know how many
blocks in this block group will be empty. The client may just wirte 1 byte, which makes most
of the blocks in this block group empty.

I think this can still be handled following similar logic of DFSOutputStream:
# A new striped block is allocated only when there is data written to it
# When the client tries to close the file, NameNode learns the last block's length from the
"block commit" request. Based on this length, the NameNode can compute the expected number
of internal blocks, and the {{completeFile}} call succeeds only when NN receives enough number
of "block received" reports from DNs.

Anyway, I will try to fix this in HDFS-8145, and we can continue the discussion there.

> Erasure coding: created util class to analyze striped block groups
> ------------------------------------------------------------------
>                 Key: HDFS-8120
>                 URL: https://issues.apache.org/jira/browse/HDFS-8120
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch, HDFS-8120.002.patch, HDFS-8120.003.patch,
HDFS-8120.004.patch, HDFS-8120.005.patch, HDFS-8120.006.patch
> The patch adds logic of calculating size of individual blocks in a striped block group.

This message was sent by Atlassian JIRA

View raw message