Mailing-List: contact chukwa-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-dev@hadoop.apache.org
Message-ID: <307030404.1245087727502.JavaMail.jira@brutus>
Date: Mon, 15 Jun 2009 10:42:07 -0700 (PDT)
From: "Jerome Boulon (JIRA)" <jira@apache.org>
To: chukwa-dev@hadoop.apache.org
Subject: [jira] Commented: (CHUKWA-185) ability to tail a whole directory
In-Reply-To: <1880458178.1240525590372.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CHUKWA-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719664#action_12719664 ] 

Jerome Boulon commented on CHUKWA-185:
--------------------------------------


Ari, it will be good to have a better control on TerminatorThread ... maybe a pool of TerminatorThread instead of creating a new one every time. A simpler solution will be to limit the number of "running" TerminatorThread's instances...

Also I'm not sure if the solution could so simple. 

If the agent crash, it shouldn't resend something that has already been sent.
Here what I was thinking of:
- make the timeWindow mandatory, could default to XX minutes
- keep track of all files that are in the processing window ( file.lastModifiedDate > now -  timeWindow), using a simple text file, (tracking file)
- when the last modified date for a file exceed the timeWindow then:
--->  do a shutdown on the adaptor for this file's entry
--->  delete the file's entry from the tracking file
- keep the tracking file in a chukwa directory and reload it at agent re-start to avoid sending the same file twice

How do you stop tailing a file? We cannot assume that we can delete a file so we need to have that built in. My proposal is to use the last modified date and the timeWindow to automatically remove adaptors.


> ability to tail a whole directory
> ---------------------------------
>
>                 Key: CHUKWA-185
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-185
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.1.2, 0.2.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.3.0
>
>         Attachments: CHUKWA-185.patch
>
>
> Right now, FileTailingAdaptors watch particular files.   It'd be great to be able to watch a whole path: to say something like /var/logs/*, where new logs created in that directory get picked up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.