hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7661) [umbrella] support hflush and hsync for erasure coded files
Date Tue, 05 Apr 2016 20:14:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227058#comment-15227058
] 

Mingliang Liu commented on HDFS-7661:
-------------------------------------

Thanks for proposing solutions to this problem.

I agree with [~jingzhao] that there are still challenges that should be addressed even if
we keep non-full cellBuffers on the "parity DN".  Moreover, for different stripe, we have
to maintain multiple cellBuffers on the "parity DN" side as the "append" operation only happens
in the same stripe. If the writer fails during the hflush operation, we have to  make sure
last flushed cellBuffer is still available, which may be in a previous (different) stripe.
If we manage all cellBuffers across different stripes in one file, we also need some kind
of "undo" mechanism for rolling back to last flushed data to handle failure. Otherwise, managing
those individual files may be nontrivial.

> [umbrella] support hflush and hsync for erasure coded files
> -----------------------------------------------------------
>
>                 Key: HDFS-7661
>                 URL: https://issues.apache.org/jira/browse/HDFS-7661
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: GAO Rui
>         Attachments: EC-file-flush-and-sync-steps-plan-2015-12-01.png, HDFS-7661-unitTest-wip-trunk.patch,
HDFS-7661-wip.01.patch, HDFS-EC-file-flush-sync-design-v20160323.pdf, HDFS-EC-file-flush-sync-design-version1.1.pdf
>
>
> We also need to support hflush/hsync and visible length. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message