incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: log rotation
Date Wed, 27 Jan 2010 21:31:08 GMT
we actually keep two different sequence numbers -- one starting since
the beginning of time and reflecting the actual sequence number of the
next chunk, and the other specifying what sequence number of the first
byte in the physical file is.

On Wed, Jan 27, 2010 at 1:27 PM, Bill Graham <billgraham@gmail.com> wrote:
> I think I have a lapse in my understanding then regarding the sequence
> number. I thought that was a pointer to the current offset in the file for
> the data that has been read. It sounds like that's not necessarily the case.
> If the sequence number continues to grow without reset after rotation, then
> how would the adapter know where to start tailing a file if the agent was
> bounced. The sequence number would be larger than the size of the file.
>
>
> On Wed, Jan 27, 2010 at 1:11 PM, Ariel Rabkin <asrabkin@gmail.com> wrote:
>>
>> Chukwa *should* keep the sequence number growing monotonically when
>> files rotate. There is some test coverage for this, but my hunch is
>> that there are corner cases we haven't got right.
>>
>> On Wed, Jan 27, 2010 at 1:08 PM, Bill Graham <billgraham@gmail.com> wrote:
>> > This thread reminded me of another related question I've been meaning to
>> > ask. My implementation is similar to the one you describe Ari, where we
>> > tail
>> > a single file indefinitely, but periodically it gets rotated by the
>> > Linux
>> > logrotate script, which handles cleanup of old files (I don't think
>> > Chukwa
>> > should handle this necessarily).
>> >
>> > My question is how does the agent handle the character offset marker
>> > when a
>> > file it's tailing gets truncated during rotation? Does it just reset it
>> > back
>> > to 0 when it sees that the file suddenly has no content?
>> >
>> >
>> > On Wed, Jan 27, 2010 at 10:48 AM, Ariel Rabkin <asrabkin@gmail.com>
>> > wrote:
>> >>
>> >> Suppose you have a file, foo, with a FileTailingAdaptor watching it.
>> >> After rotation, you'll have a foo.1 and then another foo.  The
>> >> FileTailingAdaptor will look at the latter.
>> >>
>> >> After those files have been collected, it's safe to delete them.
>> >> Right now, Chukwa doesn't do this itself. Should it?
>> >>
>> >> --Ari
>> >>
>> >> On Wed, Jan 27, 2010 at 10:18 AM, Corbin Hoenes <corbin@tynt.com>
>> >> wrote:
>> >> > I've noticed some chatter about log rotation and FileTailingAdaptor:
>> >> >  http://issues.apache.org/jira/browse/CHUKWA-421
>> >> >
>> >> > Is there some kind of documentation on how it deals with say a
>> >> > directory
>> >> > of files that get rotated.  At first glance it looks like it keeps
an
>> >> > adaptor open forever and there doesn't appear to be a way to know
>> >> > that a
>> >> > file has been processed by the adaptor so we can't really move files
>> >> > out,
>> >> > etc...  Right now we are piping apache access logs through rotatelogs
>> >> > and
>> >> > then using chukwa to tail these files.  Is there a better way?
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Ari Rabkin asrabkin@gmail.com
>> >> UC Berkeley Computer Science Department
>> >
>> >
>>
>>
>>
>> --
>> Ari Rabkin asrabkin@gmail.com
>> UC Berkeley Computer Science Department
>
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message