chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <>
Subject [jira] Commented: (CHUKWA-204) Agent side file expiration/removal
Date Mon, 18 May 2009 03:49:45 GMT


Ari Rabkin commented on CHUKWA-204:

My proposal is to bundle this functionality into an Adaptor -- I believe Adaptors have, or
should have, enough control over the Agent process to look through the list of current adaptors,
and stop the ones that need stopping.  And making this an Adaptor means that we can reuse
the existing checkpointing and adaptor configuration mechanisms.

> Agent side file expiration/removal
> ----------------------------------
>                 Key: CHUKWA-204
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection, Data Processors
>            Reporter: Jerome Boulon
>            Priority: Critical
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of daemon running
and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version will be needed.
> The advanced version should be able to track data completeness based on CHUKWA-203 and
only remove file that has been completely sent over or generate alerts.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message