chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <>
Subject [jira] Commented: (CHUKWA-421) new logfile rotation detection scheme
Date Thu, 03 Dec 2009 20:37:20 GMT


Ari Rabkin commented on CHUKWA-421:

My idea is as follows:

1) Adaptors should keep track of the last-modified date of the file they're tailing. 
2) Each time through, get the list of files whose names match the file in question, ignoring
trailing stuff after a final dot.  
    (Possible rotation results)
3) Process them, oldest-first.    Break ties using longest-first, or some name-based heuristic.

The end goal of this is that if file Foo is being tailed, and then when we look next, Foo
is shorter and there's a Foo.1, we'll finish tailing Foo.1, and then start tailing Foo. 

> new logfile rotation detection scheme
> -------------------------------------
>                 Key: CHUKWA-421
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: Improvement
>          Components: data collection
>            Reporter: Ari Rabkin
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles,
doesn't work correctly across reboots, and I think we can do better.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message