chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <>
Subject [jira] Commented: (CHUKWA-203) Track data loading from agent
Date Wed, 10 Jun 2009 07:15:08 GMT


Ari Rabkin commented on CHUKWA-203:

I don't entirely understand the scope of this.  I had thought our model was that adaptors
conceal rotation from stages farther up the line?

I definitely like the idea of issuing an "end of file, adaptor has deregistered" chunk/marker.

> Track data loading from agent
> -----------------------------
>                 Key: CHUKWA-203
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection, Data Processors
>            Reporter: Jerome Boulon
>            Priority: Critical
> Chukwa needs to track progress on all files for completeness reason. 
> The  first step could be to send adaptor information to the backend for postprocess/storage.

> This could be done at the same time of the writing checkpoint file by building a chunk
and post it to the queue.
> In addition to that, we need to track all Add/Remove operations and the final offset
for all files, the easiest way to do this will be to generate this information at the beginning
and the end of each adaptor.
> Based on that, we should be able to:
> - track any file from the add to the remove, 
> - validate that all data has been sent 
> - track all files' rotation.
> - record any permission issue (expiration policy)
> - generate alerts

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message