Mailing-List: contact chukwa-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=kmzV5vVRQ3rILW39MHZhLELkrB0hNE+oYgH7fSZfqnUnB7328SgdcL47Y17CUV/v
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Wed, 27 Jan 2010 10:42:07 -0800
Subject: Re: log rotation
From: Eric Yang <eyang@yahoo-inc.com>
To: <chukwa-user@hadoop.apache.org>
Message-ID: <C785C8FF.60E8%eyang@yahoo-inc.com>
Thread-Topic: log rotation
Thread-Index: AcqfgG0wZ6A8whx+j0WVjtIRnYq0+w==
In-Reply-To: <F5C22EA4-F2CB-4E47-88E1-9A7F7A1FB2CB@tynt.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

Hi Corbin,

Chukwa is intended for your use case, but it's not perfect in term of
functionality.  There are two possible solutions to refine Chukwa to do what
you need.  

First solution, extend UTF8 File Tailing Adaptor to delete the processed
file, and use that to tail the apache log file.  This method should be
fairly easy to implement.

Second solution, write an adaptor to receive log from named pipe, and
configure Apache to pipe the log to a perl script which writes to the named
pipe.  This method is great for reducing disk load for the webserver.  It is
more tricky to get it right.

Hope this helps.

Regards,
Eric

On 1/27/10 10:18 AM, "Corbin Hoenes" <corbin@tynt.com> wrote:

> I've noticed some chatter about log rotation and FileTailingAdaptor:
> http://issues.apache.org/jira/browse/CHUKWA-421
> 
> Is there some kind of documentation on how it deals with say a directory of
> files that get rotated.  At first glance it looks like it keeps an adaptor
> open forever and there doesn't appear to be a way to know that a file has been
> processed by the adaptor so we can't really move files out, etc...  Right now
> we are piping apache access logs through rotatelogs and then using chukwa to
> tail these files.  Is there a better way?
>  
> 
>