hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodney O'Donnell" <r...@rodojojo.com>
Subject Re: Append supported in hadoop 1.0.x branch?
Date Mon, 21 May 2012 09:29:32 GMT
Thanks again for your response, one more clarification though.

Are there any conditions under which I can trust append to work?

For example, if I use ZK to lock the hdfs file to ensure there are no
concurrent writes, then sync & close the file after each write?
Also, I assume this has nothing to do with file formats (was a little
confuses by one of the links below) and that append should not be trusted
even when using a simple text file.

Finally, any thoughts on the comment here
http://hbase.apache.org/book/hadoop.html :

    Ignore the chicken-little comment you'll find in the
hdfs-default.xmlin the description for the
dfs.support.append configuration; it says it is not enabled because there
    are “... bugs in the 'append code' and is not supported in any
production cluster.”. This comment is stale, from another era, and while
I'm sure there are bugs, the
    sync/append code has been running in production at large scale deploys
and is on by default in the offerings of hadoop by commercial vendors
] [8 <http://hbase.apache.org/book/hadoop.html#ftn.d1905e514>][9<http://hbase.apache.org/book/hadoop.html#ftn.d1905e520>

I guess this comment is only 'chicken-little' for hbase use case (i.e.,
sync is ok, append is not)?



On Fri, May 18, 2012 at 5:58 PM, Harsh J <harsh@cloudera.com> wrote:

> Rodney,
> There are two things that comprised the 0.20-append branch which added
> "append" features, which to break down simply for 1.x:
> append() - Available: Yes. Supported/Recommended: No.
> sync() - Available: Yes. Supported/Recommended: Yes.
> Please also see these links for further info/conversations on this
> topic thats happened several times before:
> https://issues.apache.org/jira/browse/HADOOP-8230
> http://search-hadoop.com/m/638TD3bAXB1
> http://search-hadoop.com/m/hBPRp1EWELS1
> Let us know if you have further questions.
> On Fri, May 18, 2012 at 12:12 PM, Rodney O'Donnell <rodo@rodojojo.com>
> wrote:
> > Hi,
> >
> > Is FileSystem.append supported on hadoop 1.0.x?  (1.0.3 in particular).
> >
> > Reading this list I thought it was back in for 1.0, but it's disabled by
> > default so I'm not 100% sure.
> > It would be great to get a definitive answer.
> >
> > Cheers,
> >
> > Rod.
> --
> Harsh J

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message