hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: Is append allowed in HDFS?
Date Tue, 10 Apr 2012 05:12:27 GMT
Harsh,

idea is to call sync for a configured batch. Still under implementation as
other parts of the system's aren't complete.

recovery/resume-from-errors-at-DN code around general tail-like
>>This sounds promising, can you please shed some more light on this.

- inder
On Tue, Apr 10, 2012 at 1:07 AM, Harsh J <harsh@cloudera.com> wrote:

> Your approach looks fine to me. I'd throw in some
> recovery/resume-from-errors-at-DN code around general tail-like
> consumption but I think you may have already done that :)
>
> But just for my curiosity - do you call sync for every record/unit or
> batch it by a few, for your problem?
>
> On Mon, Apr 9, 2012 at 10:34 PM, Inder Pall <inder.pall@gmail.com> wrote:
> > Yes makes sense. My use-case is more like a producer/consumer and
> consumer
> > trying to stream data as it arrives.
> > Has anyone hit this before and if so resolved it in a better way.
> >
> > Apologies, if i am digressing from the subject of this thread however
> seems
> > to land in the bucket of append support in HDFS.
> >
> > - Inder
> >
> >
> > On Mon, Apr 9, 2012 at 6:27 PM, Harsh J <harsh@cloudera.com> wrote:
> >>
> >> Inder,
> >>
> >> Yes, that is a requirement for readers of sync-ing data. The new meta
> >> entries can only be read by new readers. The read code would end up
> >> being exactly like the implementation for method "fs -tail" at
> >>
> >>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/fs/FsShell.java?view=markup
> >> (Line 1101)
> >>
> >> HBase does not read the WAL (HLog) continuously/vigorously as it
> >> syncs, by the way. It only reads the them when a specific request is
> >> made (for splitting, replaying and debug-printing).
> >>
> >> On Mon, Apr 9, 2012 at 6:05 PM, Inder Pall <inder.pall@gmail.com>
> wrote:
> >> > Based on what i have tried, after a sync you need to open a new
> Reader.
> >> > Please correct if that's not the write semantics.
> >> >
> >> > Thanks,
> >> > - Inder
> >> >
> >> >
> >> > On Mon, Apr 9, 2012 at 4:23 PM, Harsh J <harsh@cloudera.com> wrote:
> >> >>
> >> >> I'd also like to note that there are some unresolved issues with the
> >> >> append version in the 1.x (stable) line.
> >> >>
> >> >> Note that HBase's use of the 0.20-append branch features are limited
> >> >> to using "sync" calls alone (Described in p68 "Coherency Model",
> >> >> Chapter 3 (The Hadoop Distributed File System) in Hadoop: The
> >> >> Definitive Guide, 2nd Edition (O'Reilly)). Not the file-reopening
> >> >> "append" calls. The latter is what is still with issues in the 1.x
> >> >> releases today. Using the former is alright if its done in the way
> >> >> similar to HBase's WAL (HLog) (or for similar needs).
> >> >>
> >> >> On Mon, Apr 9, 2012 at 3:45 PM, Ioan Eugen Stan <
> stan.ieugen@gmail.com>
> >> >> wrote:
> >> >> > 2012/4/7 Florin P <florinpico@yahoo.com>:
> >> >> >> Hello!
> >> >> >>   Just google it for supporting of append into HDFS files
and the
> >> >> >> result:
> >> >> >> I'm puzzled. Can someone say: YES you can append in TextFile
or
> >> >> >> SequenceFile
> >> >> >> or whatever format. If yes, in which version this feature
is
> >> >> >> supported
> >> >> >> ?
> >> >> >> Also where can I find a good example of using the API? I know
that
> >> >> >> is a
> >> >> >> long
> >> >> >> debate about this subject, but really it is challenge to find
on
> the
> >> >> >> google
> >> >> >> the current status of this feature.
> >> >> >> I look forward for a trust source answer.
> >> >> >> Thank you,
> >> >> >>   Regards,
> >> >> >>    Florin
> >> >> >
> >> >> > Hi Florian,
> >> >> >
> >> >> > HDFS supports append in Hadoop 1.0.x branch and also 0.22 (a.k.a
> >> >> > hadoop 2.x branch) and 0.23 (a.k.a hadoop 3.x branch).
> >> >> >
> >> >> > [1] http://hbase.apache.org/book/hadoop.html
> >> >> > [2] http://hbase.apache.org/book/hadoop.html -- search for append
> in
> >> >> > release notes
> >> >> >
> >> >> > Cheers,
> >> >> > --
> >> >> > Ioan Eugen Stan
> >> >> > http://ieugen.blogspot.com/
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks,
> >> > - Inder
> >> >   Tech Platforms @Inmobi
> >> >   Linkedin - http://goo.gl/eR4Ub
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
> >
> >
> > --
> > Thanks,
> > - Inder
> >   Tech Platforms @Inmobi
> >   Linkedin - http://goo.gl/eR4Ub
>
>
>
> --
> Harsh J
>



-- 
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

Mime
View raw message