hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
Date Tue, 22 Sep 2015 23:54:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jing Zhao updated HDFS-9040:
    Attachment: HDFS-9040-HDFS-7285.004.patch

Update the patch based on our discussion. The main changes compared with 003 version:
# Flush out all the enqueued data to DataNodes before handling failures and bumping GS
# Instead of let each DataStreamer write their own last empty packet of the block, we do it
in the StripedOutputStream level so that we can still bump GS for failure handling before
some streamers close their internal blocks.
# During the test I found that some data streamer may take a long time to close/create datanode
connections. This may cause other streamers' connections timeout. Thus the new patch adds
an upper bound for the total waiting time of creating datanode connections during failure

A big missing part is test. We need to add a lot more tests to cover all different scenarios.
Maybe we can use HDFS-9098 to do it.

> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
> -------------------------------------------------------------------------------------------
>                 Key: HDFS-9040
>                 URL: https://issues.apache.org/jira/browse/HDFS-9040
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Jing Zhao
>         Attachments: HDFS-9040-HDFS-7285.002.patch, HDFS-9040-HDFS-7285.003.patch, HDFS-9040-HDFS-7285.004.patch,
HDFS-9040.00.patch, HDFS-9040.001.wip.patch, HDFS-9040.02.bgstreamer.patch
> The general idea is to simplify error handling logic.
> Proposal 1:
> A BlockGroupDataStreamer to communicate with NN to allocate/update block, and StripedDataStreamer
s only have to stream blocks to DNs.
> Proposal 2:
> See below the [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
from [~jingzhao].

This message was sent by Atlassian JIRA

View raw message