chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-203) Track data loading from agent
Date Wed, 10 Jun 2009 07:15:08 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717960#action_12717960
] 

Ari Rabkin commented on CHUKWA-203:
-----------------------------------

I don't entirely understand the scope of this.  I had thought our model was that adaptors
conceal rotation from stages farther up the line?

I definitely like the idea of issuing an "end of file, adaptor has deregistered" chunk/marker.


> Track data loading from agent
> -----------------------------
>
>                 Key: CHUKWA-203
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-203
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection, Data Processors
>            Reporter: Jerome Boulon
>            Priority: Critical
>
> Chukwa needs to track progress on all files for completeness reason. 
> The  first step could be to send adaptor information to the backend for postprocess/storage.

> This could be done at the same time of the writing checkpoint file by building a chunk
and post it to the queue.
> In addition to that, we need to track all Add/Remove operations and the final offset
for all files, the easiest way to do this will be to generate this information at the beginning
and the end of each adaptor.
> Based on that, we should be able to:
> - track any file from the add to the remove, 
> - validate that all data has been sent 
> - track all files' rotation.
> - record any permission issue (expiration policy)
> - generate alerts
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message