hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7661) [umbrella] support hflush and hsync for erasure coded files
Date Thu, 07 Apr 2016 01:02:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229446#comment-15229446

Kai Zheng commented on HDFS-7661:

Thanks all for the great discussions and hard work. Sorry for my interruption.

The proposed way that sends all cellBuffers to all parity DNs additionally sounds to work
because all the real data in question or to be flushed are replicated to the parity DNs thus
the parity cells can be computed on DNs on demand. On the other hand, this will complicate
the effort to support advanced erasure codecs because any codec/coder will then need to aware
the trick played here. 

I understand there is no easy solution to this problem. I'm wondering the involved complexity
and overhead for the support may be too much because we don't have any practical solid use
cases that requires hflush/hsync in striping mode. Looks like HBase workload did be mentioned,
but I doubt if there is any benchmark that proves striping will be suitable for HBase. Any
HBase fellow could cast some thoughts here? How about making hflush/hsync api as NOT-SUPPORTED
or TO-BE-IMPLEMENTED for striping files? This may resolve a blocker for the 3.0 release, and
when any practical use cases or requirement for such, this effort can then be revisited.

> [umbrella] support hflush and hsync for erasure coded files
> -----------------------------------------------------------
>                 Key: HDFS-7661
>                 URL: https://issues.apache.org/jira/browse/HDFS-7661
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: GAO Rui
>         Attachments: EC-file-flush-and-sync-steps-plan-2015-12-01.png, HDFS-7661-unitTest-wip-trunk.patch,
HDFS-7661-wip.01.patch, HDFS-EC-file-flush-sync-design-v20160323.pdf, HDFS-EC-file-flush-sync-design-version1.1.pdf,
> We also need to support hflush/hsync and visible length. 

This message was sent by Atlassian JIRA

View raw message