Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates
 209.85.210.194 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=BDdmThv16LoQeuZdN5D7MDYh4x/rETM+MuWPxJCbc6NB8y9IxCaNvhyF9alHiMui8+
         Gs25k1VpM0pafLl0v0u6NsZCZvcLF7v+bgFHzhCnubNOnynrp1hLyO0CWC29guiOmuiB
         K2BUAydixT7Kxx9fbfFvERT+p7GyVX4zvADxM=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <e4858b791001111742l3a879e5dh78f0d56a0f4090b7@mail.gmail.com>
References: <31a243e71001111642l534b9424je20f08387afc1271@mail.gmail.com>
	 <4B4BCD32.2070406@yahoo-inc.com>
	 <31a243e71001111733s1d136dc8t4ab62d5df170de2a@mail.gmail.com>
	 <e4858b791001111742l3a879e5dh78f0d56a0f4090b7@mail.gmail.com>
Date: Mon, 11 Jan 2010 17:53:24 -0800
Message-ID: <31a243e71001111753h196bf6f0l986a0bb5d8834577@mail.gmail.com>
Subject: Re: Reading while appending in 0.21
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Hairong,

I now understand that this is outside of the scope of 265 (which works grea=
t).

What I'm thinking is that tailing seems achievable now that we have
hflush, tho I'm saying this without much knowledge of the internals.

J-D

On Mon, Jan 11, 2010 at 5:42 PM, Hairong Kuang <kuang.hairong@gmail.com> wr=
ote:
> This semantics is stated in our append specification attached to HDFS-265=
.
> Hflushed data are visible to new readers who open the file after the hflu=
sh.
>
> Hairong
>
> On Mon, Jan 11, 2010 at 5:33 PM, Jean-Daniel Cryans <jdcryans@apache.org>
> wrote:
>>
>> Thanks for the answer Konstantin, good to know. I also see that the
>> commenter doesn't seem pleased with the fact that the file has to be
>> reopened ;)
>>
>> So basically what I wrongly expected was a tailing feature. Searches
>> in Jira don't give me a hit, would it make sense to open one?
>>
>> In the mean time, I'm ok with recreating the reader.
>>
>> Thx,
>>
>> J-D
>>
>> On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <cos@yahoo-inc.com>
>> wrote:
>> > Jean,
>> >
>> > I believe this is how it has been intended to work. If you take a look
>> > at
>> > the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
>> > workspace you'll a comment on this particular aspect in the line 124.
>> >
>> > Hope it helps,
>> > =A0Cos
>> >
>> > On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm trying to use the new hflush function from 0.21 so that a
>> >> SequenceFile.Reader could read edits from a SequenceFile.Writer after
>> >> a signal on a Condition. If I:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >>
>> >> It works fine. But after that if I only next(), using the same reader=
,
>> >> after appending/hflush I won't see the new edits. But, if I create a
>> >> new Reader after calling hflush, it works fine.
>> >>
>> >> So this does not work:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> next() through the entries
>> >>
>> >> This does:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> create a new Reader
>> >> seek
>> >> next() through the entries
>> >>
>> >> Is that the intended behavior?
>> >>
>> >> Thx!
>> >>
>> >> J-D
>> >
>
>