incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: scale of File Tail
Date Fri, 23 Dec 2011 03:12:20 GMT
Huge file is better for file tailing adaptor actually.  FileTailing
adaptor checks every 2 seconds to see if file has rotated.  If the
rotating frequency is high, like rotating every seconds.  Then there
may be a chance of losing track of correct offset in the files.
Rotated log every couple minutes with large size, helps FileTailing
adaptor to track offset in the files correctly.
As far as throughput, it can stream up to 10MB/sec from our performance test.

regards,
Eric

On Thu, Dec 22, 2011 at 6:44 PM, AD <straightflush@gmail.com> wrote:
> hey,
>
>  does anyone know if there are any throughput issues with FileTailing
> adaptors?   If i am writing huge files is there risk of missing lines
> (especially if the files are getting rotated).  I am just trying to decide
> if running map/reduce on a rotated log every X minutes is more safe than
> FileTail if the log writes get massively high (thousands/sec)
>
> Cheers,
> AD

Mime
View raw message