chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <>
Subject [jira] Commented: (CHUKWA-185) ability to tail a whole directory
Date Mon, 15 Jun 2009 17:42:07 GMT


Jerome Boulon commented on CHUKWA-185:

Ari, it will be good to have a better control on TerminatorThread ... maybe a pool of TerminatorThread
instead of creating a new one every time. A simpler solution will be to limit the number of
"running" TerminatorThread's instances...

Also I'm not sure if the solution could so simple. 

If the agent crash, it shouldn't resend something that has already been sent.
Here what I was thinking of:
- make the timeWindow mandatory, could default to XX minutes
- keep track of all files that are in the processing window ( file.lastModifiedDate > now
-  timeWindow), using a simple text file, (tracking file)
- when the last modified date for a file exceed the timeWindow then:
--->  do a shutdown on the adaptor for this file's entry
--->  delete the file's entry from the tracking file
- keep the tracking file in a chukwa directory and reload it at agent re-start to avoid sending
the same file twice

How do you stop tailing a file? We cannot assume that we can delete a file so we need to have
that built in. My proposal is to use the last modified date and the timeWindow to automatically
remove adaptors.

> ability to tail a whole directory
> ---------------------------------
>                 Key: CHUKWA-185
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.1.2, 0.2.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.3.0
>         Attachments: CHUKWA-185.patch
> Right now, FileTailingAdaptors watch particular files.   It'd be great to be able to
watch a whole path: to say something like /var/logs/*, where new logs created in that directory
get picked up.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message