hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7661) [umbrella] support hflush and hsync for erasure coded files
Date Fri, 08 Apr 2016 12:15:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232103#comment-15232103
] 

Walter Su commented on HDFS-7661:
---------------------------------

Great design/discussion. Since we come back to discuss the use cases, and "effort vs benefit“.
I'm thinking if the use cases are rare, we can provide a simpler workaround. We provide:
1. a fake "flush", which only flushes the full stripe, and doesn't flush the last partial
stripe. It won't make sure every byte is safe, but it helps recovery logic to recover more
data.
2. a real "flush". The easiest way to do this is to start a new block group. It makes sure
the data written before the "flush" is safe and visible. It saves user the trouble of closing
and appending the same file.

Since we support variable-length blocks, it's totally doable. I need to mention that the implementation
of appending striped file also utilizes variable-length blocks. The trouble is creating too
many block groups. But if there's too many small blocks, and if they are  adjacent in the
same file, we can concatenate them to a bigger block, although striped blocks concatenation
seems not easy either.

> [umbrella] support hflush and hsync for erasure coded files
> -----------------------------------------------------------
>
>                 Key: HDFS-7661
>                 URL: https://issues.apache.org/jira/browse/HDFS-7661
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: GAO Rui
>         Attachments: EC-file-flush-and-sync-steps-plan-2015-12-01.png, HDFS-7661-unitTest-wip-trunk.patch,
HDFS-7661-wip.01.patch, HDFS-EC-file-flush-sync-design-v20160323.pdf, HDFS-EC-file-flush-sync-design-version1.1.pdf,
Undo-Log-Design-20160406.jpg
>
>
> We also need to support hflush/hsync and visible length. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message