hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3960) Snapshot of Being Written Files
Date Tue, 30 Oct 2012 17:50:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487060#comment-13487060

Aaron T. Myers commented on HDFS-3960:

bq. The document proposes to add new functionality as far as I understand. I just propose
to add a parameter to fsync(). Because sync can and should adjust file length, just like close.
The client is the only authority that knows the exact file length.

I think we're agreeing here, just using different terms. The document is presently discussing
how the DFSOutputStream API would be changed to add an optional parameter to hflush() and
hsync() which would cause the client to update the length on the NN. What you're describing
is how that would be implemented: by changing the ClientProtocol#fsync method to also take
a length parameter. I'll update the document to make this clear.

bq. I don't think files that were not hflush-ed should be excluded from snapshots as the document
proposes. Everything should be versioned.

That would be fine too. Excluding them seemed simpler, but I don't feel strongly about it.
The length of that file in the snapshot would then be at the last block boundary before the
snapshot was taken. I'll update the design document to reflect that.

Thanks a lot for the comments, Konstantin, and for taking a look at the doc. Your feedback
is very valuable.
> Snapshot of Being Written Files
> -------------------------------
>                 Key: HDFS-3960
>                 URL: https://issues.apache.org/jira/browse/HDFS-3960
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
> Here is a design question: Suppose there is a being written file when a snapshot is being
taken.  What should the length of the file be shown in the snapshot?  In other words, how
to determine the length of being written file when a snapshot is being taken?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message