hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Reading while appending in 0.21
Date Tue, 12 Jan 2010 01:53:24 GMT
Hi Hairong,

I now understand that this is outside of the scope of 265 (which works great).

What I'm thinking is that tailing seems achievable now that we have
hflush, tho I'm saying this without much knowledge of the internals.

J-D

On Mon, Jan 11, 2010 at 5:42 PM, Hairong Kuang <kuang.hairong@gmail.com> wrote:
> This semantics is stated in our append specification attached to HDFS-265.
> Hflushed data are visible to new readers who open the file after the hflush.
>
> Hairong
>
> On Mon, Jan 11, 2010 at 5:33 PM, Jean-Daniel Cryans <jdcryans@apache.org>
> wrote:
>>
>> Thanks for the answer Konstantin, good to know. I also see that the
>> commenter doesn't seem pleased with the fact that the file has to be
>> reopened ;)
>>
>> So basically what I wrongly expected was a tailing feature. Searches
>> in Jira don't give me a hit, would it make sense to open one?
>>
>> In the mean time, I'm ok with recreating the reader.
>>
>> Thx,
>>
>> J-D
>>
>> On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <cos@yahoo-inc.com>
>> wrote:
>> > Jean,
>> >
>> > I believe this is how it has been intended to work. If you take a look
>> > at
>> > the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
>> > workspace you'll a comment on this particular aspect in the line 124.
>> >
>> > Hope it helps,
>> >  Cos
>> >
>> > On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm trying to use the new hflush function from 0.21 so that a
>> >> SequenceFile.Reader could read edits from a SequenceFile.Writer after
>> >> a signal on a Condition. If I:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >>
>> >> It works fine. But after that if I only next(), using the same reader,
>> >> after appending/hflush I won't see the new edits. But, if I create a
>> >> new Reader after calling hflush, it works fine.
>> >>
>> >> So this does not work:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> next() through the entries
>> >>
>> >> This does:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> create a new Reader
>> >> seek
>> >> next() through the entries
>> >>
>> >> Is that the intended behavior?
>> >>
>> >> Thx!
>> >>
>> >> J-D
>> >
>
>

Mime
View raw message