hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
Date Mon, 25 Aug 2014 22:58:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109922#comment-14109922
] 

Arpit Agarwal commented on HDFS-6898:
-------------------------------------

Thanks for looking at the patch.

I am not sure whether the Linux implementation of {{fallocate}} results in extra file I/O.
However we still have the other issue that Colin mentioned - we cannot deduce the end of file
on restart since our block file format lacks any header/meta information.

For now we can just "reserve" space for a full block when an RBW block is created. The "reserved"
space is progressively reduced as bytes are ack'ed to the client. Any remaining space is released
when the block is finalized.

[~cmccabe], the attached .03.patch is current.

> DN must reserve space for a full block when an RBW block is created
> -------------------------------------------------------------------
>
>                 Key: HDFS-6898
>                 URL: https://issues.apache.org/jira/browse/HDFS-6898
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.5.0
>            Reporter: Gopal V
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
>
>
> DN will successfully create two RBW blocks on the same volume even if the free space
is sufficient for just one full block.
> One or both block writers may subsequently get a DiskOutOfSpace exception. This can be
avoided by allocating space up front.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message