hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1700) Append to files in HDFS
Date Thu, 06 Sep 2007 08:30:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525365
] 

eric baldeschwieler commented on HADOOP-1700:
---------------------------------------------

Why would the name node not keep the timestamp/version?  Seems to me it will need it to disambiguate
new block reports and such.  We can afford the bytes IMO.

Those statements about "reasonably synchronized clocks" worry me.  Especially if clients or
datanodes are producing them.  If the name node is not the source of truth, I'm afraid we
can introduce weird cases.

I think the next logical step here is for us to outline a more detail proposal for review.
 I think this is going to take a couple of distinct steps.  We can outline those and get to
agreement on the basics.

Jim's request that a file exist if its producer dies before a close sounds like it would be
easy, but then we get to flush semantics...  I take it you don't just want the last complete
128M block Jim?  If you want the last "log line" then you need something similar to what I
outlined.



> Append to files in HDFS
> -----------------------
>
>                 Key: HADOOP-1700
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1700
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: stack
>
> Request for being able to append to files in HDFS has been raised a couple of times on
the list of late.   For one example, see http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193.
 Other mail describes folks' workarounds because this feature is lacking: e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480
(Later on this thread, Jim Kellerman re-raises the HBase need of this feature).  HADOOP-337
'DFS files should be appendable' makes mention of file append but it was opened early in the
life of HDFS when the focus was more on implementing the basics rather than adding new features.
 Interest fizzled.  Because HADOOP-337 is also a bit of a grab-bag -- it includes truncation
and being able to concurrently read/write -- rather than try and breathe new life into HADOOP-337,
instead, here is a new issue focused on file append.  Ultimately, being able to do as the
google GFS paper describes -- having multiple concurrent clients making 'Atomic Record Append'
to a single file would be sweet but at least for a first cut at this feature, IMO, a single
client appending to a single HDFS file letting the application manage the access would be
sufficent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message