hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6633) Support reading new data in a being written file until the file is closed
Date Thu, 10 Jul 2014 08:06:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057249#comment-14057249

Jonathan Hsieh commented on HDFS-6633:

Here's one for hbase: HBase uses a hdfs files as a write ahead log that is periodically "rolled"
(close one file, open the next).  In the even of a machine failure, we need to wait for the
dead machine's log file to close before we can read all of its contents and get back to a
consistent state.  If we had a hdfs tail functionality, standby nodes could be reading data
as it is being written from an alternate replica and thus make recovery faster.

> Support reading new data in a being written file until the file is closed
> -------------------------------------------------------------------------
>                 Key: HDFS-6633
>                 URL: https://issues.apache.org/jira/browse/HDFS-6633
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h6633_20140707.patch, h6633_20140708.patch
> When a file is being written, the file length keeps increasing.  If the file is opened
for read, the reader first gets the file length and then read only up to that length.  The
reader will not be able to read the new data written afterward.
> We propose adding a new feature so that readers will be able to read all the data until
the writer closes the file.

This message was sent by Atlassian JIRA

View raw message