Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of asrabkin@gmail.com designates
 209.85.214.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAM2WPyy7odE8bQOXAQBx-KTqKvLF=mifAArJ5sa1toLhAz4_9w@mail.gmail.com>
References: 
 <CAM2WPywKhprgUJkUz7dDbk=pacAKoZPcOqHxx7eJE2NN=bpMQQ@mail.gmail.com>
	<CAM2WPyy7odE8bQOXAQBx-KTqKvLF=mifAArJ5sa1toLhAz4_9w@mail.gmail.com>
Date: Tue, 15 May 2012 23:51:38 -0700
Message-ID: 
 <CAJcOcz4Pp1HhV7-QjDSAfYidMXKJbwHDVUTYLUULdRu=_eYUpg@mail.gmail.com>
Subject: Re: the check point offset is bigger than the log file size
From: Ariel Rabkin <asrabkin@gmail.com>
To: chukwa-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Rotation is a bit of a mess.

We've tried a couple strategies to handle it, none of which are perfect.
One approach is to have a modified logger that explicitly invokes
chukwa, starting and stopping adaptors.
The other is that the FileTailingAdaptors keep not only a physical
"how long is the file" offset, but a logical "what is the byte number
of the first byte of the file" -- the idea is that if the file
rotates, the adaptor should add the length of the rotated-out section
to the length of the current file.

This is a bit fragile, since the adaptor has to guess which was the
previously-rotated file. I believe we use timestamps for that. I
suspect it won't always work.

--Ari

On Tue, May 15, 2012 at 11:45 PM, IvyTang <ivytang0812@gmail.com> wrote:
> =A0 =A0 After reading the source code ,i'm confuesd about the checkpoint =
file .
>
> =A0 =A0 The file tailer generate the chunks into the memlimitqueue, the
> httpsender get the chunks to send from the=A0 memlimitqueue. And after th=
e
> httpsender send the chunks to collector=A0reliably ,the=A0reportCommit(Ad=
aptor
> src, long uuid) will be called.
>
> =A0=A0 In this=A0reportCommit(Adaptor src, long uuid) method, the src is =
the
> adaptor , the uuid is the offset of those chunks which have beend in the
> file .And if the uuid is > =A0adaptor.offset , the means some chunks have=
 been
> sent , so the=A0adaptor.offset is assigned to the uuid.
>
> =A0 This works file when the log file is =A0not=A0rotating .
>
> =A0 =A0 But if the log file is rotating(i mean the way like log4j , move =
this
> file to *.1 and generate a file named *), the=A0=A0adaptor.offset is the =
offset
> of those chunks been sent in last file , it's of course very big . but uu=
id
> is the offset of chunks been sent of this file , the uuid is smaller the
> the=A0adaptor.offset=A0.
>
> =A0 =A0 So the checkpoint file won't change .
>
> =A0 =A0 Even though chukwa is still sending chunks to collector , but if =
chukwa
> restarted , the checkpoint is larger than the log file size , the log fil=
e
> will be sent again.
>
>
>
> On Mon, May 14, 2012 at 7:01 PM, IvyTang <ivytang0812@gmail.com> wrote:
>>
>> The gamelog size is=A0158023223, but the check point file is
>>
>> ADD adaptor_2963225a90653a309cf779d4a1d815a3 =3D
>> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTaili=
ngAdaptorUTF8
>> Gamelog 0 /var/log/dataproxy/gamelog 229406124
>>
>> The gamelog didn't rotate , i'm sure.
>>
>> But the check point file size is bigger than the file size , the chukwa
>> WARN Thread-2 FileTailingAdaptor -
>> Adaptor|adaptor_2963225a90653a309cf779d4a1d815a3| file:
>> /var/log/dataproxy/gamelog, has rotated and no detection - reset counter=
s to
>> 0L
>> And the agent began to transfer the whole log file.
>>
>> I just feel confused why agent generate a offset size is bigger than the
>> log size when the gamelog did not rotate.
>>
>> The chukwa version is 0.4.0
>>
>> --
>> Best regards,
>>
>> Ivy Tang
>>
>>
>>
>
>
>
> --
> Best regards,
>
> Ivy Tang
>
>
>


--=20
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department