flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: using flume to manage all log rotation
Date Thu, 10 Nov 2011 03:46:02 GMT
Hi Andrew,

We use logrotate to rotate, gzip, and purge old log files.
We also use tail -F from Flume because Flume's own tail has its issues (discussed a lot on
the old Flume ML 6 or more months ago)

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/

>From: Andrew McNair <andrew.mcnair@gmail.com>
>To: flume-user@incubator.apache.org
>Sent: Wednesday, November 9, 2011 7:10 PM
>Subject: using flume to manage all log rotation
>I'm interested in using flume to manage all log rotation in my deployment. Currently flume
is tailing a logfile. Every night log4j creates a gz backup of the previous day's logfile.
There's a cron job to delete older gz backup files.
>The problems with this approach are:
>- Log4j's (apache-log4j-extras's) RollingFileAppender isn't super robust. A few days ago
it locked up all threads while it gzipped the logfiles
>- I need a strategy to handle the flume agent going down and missing events because log4j
has rotated the file flume is tailing
>- Once Flume has delivered the content of the logfile I don't need the gz backups, so
they're just wasting space
>What I'd really like is to have flume handle rolling over the original files, and deleting
backup files once flume is fairly confident it's delivered the contents to the source. Has
anybody setup something like this? Any pointers would be much appreciated. 
> Andrew   
View raw message