hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florin P <florinp...@yahoo.com>
Subject Re: Is append allowed in HDFS?
Date Fri, 13 Apr 2012 09:56:57 GMT
Hello!
 Thank you all for all responses. It is possible to have a matrix of hadoop file input format
that supports append or if I understood correctly, all formats are now supporting append?
Thanks a lot.
  Regards,
 Florin



________________________________
 From: Inder Pall <inder.pall@gmail.com>
To: hdfs-user@hadoop.apache.org 
Sent: Tuesday, April 10, 2012 8:12 AM
Subject: Re: Is append allowed in HDFS?
 

Harsh,

idea is to call sync for a configured batch. Still under implementation as other parts of
the system's aren't complete.

recovery/resume-from-errors-at-DN code around general tail-like
>>This sounds promising, can you please shed some more light on this.

- inder

On Tue, Apr 10, 2012 at 1:07 AM, Harsh J <harsh@cloudera.com> wrote:

Your approach looks fine to me. I'd throw in some
>recovery/resume-from-errors-at-DN code around general tail-like
>consumption but I think you may have already done that :)
>
>But just for my curiosity - do you call sync for every record/unit or
>batch it by a few, for your problem?
>
>
>On Mon, Apr 9, 2012 at 10:34 PM, Inder Pall <inder.pall@gmail.com> wrote:
>> Yes makes sense. My use-case is more like a producer/consumer and consumer
>> trying to stream data as it arrives.
>> Has anyone hit this before and if so resolved it in a better way.
>>
>> Apologies, if i am digressing from the subject of this thread however seems
>> to land in the bucket of append support in HDFS.
>>
>> - Inder
>>
>>
>> On Mon, Apr 9, 2012 at 6:27 PM, Harsh J <harsh@cloudera.com> wrote:
>>>
>>> Inder,
>>>
>>> Yes, that is a requirement for readers of sync-ing data. The new meta
>>> entries can only be read by new readers. The read code would end up
>>> being exactly like the implementation for method "fs -tail" at
>>>
>>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/fs/FsShell.java?view=markup
>>> (Line 1101)
>>>
>>> HBase does not read the WAL (HLog) continuously/vigorously as it
>>> syncs, by the way. It only reads the them when a specific request is
>>> made (for splitting, replaying and debug-printing).
>>>
>>> On Mon, Apr 9, 2012 at 6:05 PM, Inder Pall <inder.pall@gmail.com> wrote:
>>> > Based on what i have tried, after a sync you need to open a new Reader.
>>> > Please correct if that's not the write semantics.
>>> >
>>> > Thanks,
>>> > - Inder
>>> >
>>> >
>>> > On Mon, Apr 9, 2012 at 4:23 PM, Harsh J <harsh@cloudera.com> wrote:
>>> >>
>>> >> I'd also like to note that there are some unresolved issues with the
>>> >> append version in the 1.x (stable) line.
>>> >>
>>> >> Note that HBase's use of the 0.20-append branch features are limited
>>> >> to using "sync" calls alone (Described in p68 "Coherency Model",
>>> >> Chapter 3 (The Hadoop Distributed File System) in Hadoop: The
>>> >> Definitive Guide, 2nd Edition (O'Reilly)). Not the file-reopening
>>> >> "append" calls. The latter is what is still with issues in the 1.x
>>> >> releases today. Using the former is alright if its done in the way
>>> >> similar to HBase's WAL (HLog) (or for similar needs).
>>> >>
>>> >> On Mon, Apr 9, 2012 at 3:45 PM, Ioan Eugen Stan <stan.ieugen@gmail.com>
>>> >> wrote:
>>> >> > 2012/4/7 Florin P <florinpico@yahoo.com>:
>>> >> >> Hello!
>>> >> >>   Just google it for supporting of append into HDFS files
and the
>>> >> >> result:
>>> >> >> I'm puzzled. Can someone say: YES you can append in TextFile
or
>>> >> >> SequenceFile
>>> >> >> or whatever format. If yes, in which version this feature is
>>> >> >> supported
>>> >> >> ?
>>> >> >> Also where can I find a good example of using the API? I know
that
>>> >> >> is a
>>> >> >> long
>>> >> >> debate about this subject, but really it is challenge to find
on the
>>> >> >> google
>>> >> >> the current status of this feature.
>>> >> >> I look forward for a trust source answer.
>>> >> >> Thank you,
>>> >> >>   Regards,
>>> >> >>    Florin
>>> >> >
>>> >> > Hi Florian,
>>> >> >
>>> >> > HDFS supports append in Hadoop 1.0.x branch and also 0.22 (a.k.a
>>> >> > hadoop 2.x branch) and 0.23 (a.k.a hadoop 3.x branch).
>>> >> >
>>> >> > [1] http://hbase.apache.org/book/hadoop.html
>>> >> > [2] http://hbase.apache.org/book/hadoop.html -- search for append
in
>>> >> > release notes
>>> >> >
>>> >> > Cheers,
>>> >> > --
>>> >> > Ioan Eugen Stan
>>> >> > http://ieugen.blogspot.com/
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Harsh J
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Thanks,
>>> > - Inder
>>> >   Tech Platforms @Inmobi
>>> >   Linkedin - http://goo.gl/eR4Ub
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>>
>> --
>> Thanks,
>> - Inder
>>   Tech Platforms @Inmobi
>>   Linkedin - http://goo.gl/eR4Ub
>
>
>
>--
>Harsh J
>


-- 
Thanks,
- Inder 
 
Tech Platforms @Inmobi 
 
Linkedin - http://goo.gl/eR4Ub
Mime
View raw message